Module 07: Distributed Systems
Prerequisites
- Complete GPU Kernels and Compilers.
- Understand bandwidth, latency, synchronization, gradients, and optimizer state.
Outcomes
- Derive collective semantics and first-order communication costs.
- Match data, tensor, pipeline, and sharded parallelism to constraints.
- Design observability and recovery for distributed correctness and performance.
Ordered free resources
- PyTorch Distributed Overview · Tier A · Documentation / tutorials / code — learn process groups, collectives, and training APIs.
- NVIDIA Collective Communications Library Documentation · Tier A · Documentation / reference — inspect GPU collective semantics and topology concerns.
- Deep Learning Systems · Tier A · Course / video / labs — connect communication with computation graphs.
- Stanford CS336 — Language Modeling from Scratch · Tier A · Course / notes / assignments — place parallel training choices inside a complete language-model system.
Checkpoints
- Complete Distributed All-Reduce Reasoning.
- Produce a communication-volume estimate for two parallelism strategies.
- Write a triage tree for hang, out-of-memory, and divergence failures.
Self-tests
- Why is all-reduce not equivalent to gathering every full tensor on one rank?
- Which topology details can invalidate a simple bandwidth estimate?
- Where can communication overlap with useful computation?
Capstone
Design a multi-GPU plan for a model that does not fit on one device. Include tensor placement, collectives, topology assumptions, memory and communication budgets, failure recovery, and a small-scale validation experiment.
Next: Evaluation