Introduction to PyTorch for HPC

Overview

This short course provides a practical introduction to building artificial neural networks using PyTorch, a powerful and flexible deep l earning framework. The course covers the fundamentals of PyTorch, including tensors, automatic differentiation, and model building. Participants will learn how to construct, train, and optimize neural networks while exploring real-world applications. The course also includes best practices for experiment tracking and project setup. By the end of the course, participants will have a solid foundation in PyTorch and be able to develop deep learning models for a variety of tasks.

Table of Contents

1. Introduction to PyTorch

• Background
• Overview of PyTorch and its core components
• Advantages of PyTorch

2. Building a Simple Neural Network

• Neural Network Construction
• Training and Testing
• Validation Sets

3. GPU Acceleration for Deep Learning

• Why use GPUs?
• PyTorch and GPUs
• Best practices for usig GPUs
• Writing SLURM scripts for PyTorch jobs

4.Transfer Learning

• What is Transfer Learning?
• Loading and Preprocessing Datasets
• Leaveraging Pre-Trained Models for New Tasks

5. Optimizing Neural Networks

• Optimizers and learning rate adjustments
• Techniques for avoiding overfitting (dropout, batch normalization)
• Validation Sets
• Data augmentation techniques
• Handling imbalanced data

6. PyTorch Experimentation and Project Setup

• Structuring a PyTorch project
• Managing experiments with logging frameworks (TensorBoard, Weights & Biases)
• Creating reproducible experiments
• Using torch.utils.data.DataLoader for efficient batch processing
• Debugging PyTorch models

This course is not an introduction to deep learning. It will not cover deep learning theory or mathematical concepts. This course will cover how to develop ANNs and CNNs. You are strongly encouraged to repeat the examples and do the exercises.

Prerequisites

• Deep Learning Theory
• Python Programming