Multi-GPU Training Lab

Overview

The Multi-GPU Training Lab is our most powerful training environment — two NVIDIA A100 GPUs with 40 GB of VRAM each, connected via NVLink for ultra-fast inter-GPU communication. This is the same class of hardware used by AI research labs to train state-of-the-art models. You'll use this lab for distributed training exercises: scaling a training job across multiple GPUs using PyTorch FSDP or DeepSpeed, pre-training small transformer language models, training large diffusion models, and running compute-intensive research experiments. These labs are reserved for advanced course exercises where single-GPU training would be impractical.

What You'll Do in This Lab

• Pre-train transformer language models (GPT-2 scale, 124M+ params) from scratch
• Scale training across 2 GPUs using PyTorch DDP and FSDP
• Train diffusion models (DDPM) for image generation
• Run mixed-precision training with BF16 on A100 hardware
• Implement DeepSpeed ZeRO stages for memory-efficient training
• Benchmark single-GPU vs multi-GPU throughput and scaling efficiency

Lab Workflow