Module 07: Distributed Systems
Prerequisites
- Complete GPU Kernels and Compilers.
- Understand bandwidth, latency, synchronization, gradients, and optimizer state.
Outcomes
- Derive collective semantics and first-order communication costs.
- Match data, tensor, pipeline, and sharded parallelism to constraints.
- Design observability and recovery for distributed correctness and performance.
Ordered free resources
- PyTorch Distributed Documentation (
pytorch-distributed) — learn process groups, collectives, and training APIs. - NCCL Documentation (
nccl-docs) — inspect GPU collective semantics and topology concerns. - Deep Learning Systems Course (
dlsyscourse) — connect communication with computation graphs. - Full Stack Deep Learning (
fsdl) — place distributed training inside operational practice.
Checkpoints
- Complete Distributed All-Reduce Reasoning.
- Produce a communication-volume estimate for two parallelism strategies.
- Write a triage tree for hang, out-of-memory, and divergence failures.
Self-tests
- Why is all-reduce not equivalent to gathering every full tensor on one rank?
- Which topology details can invalidate a simple bandwidth estimate?
- Where can communication overlap with useful computation?
Capstone
Design a multi-GPU plan for a model that does not fit on one device. Include tensor placement, collectives, topology assumptions, memory and communication budgets, failure recovery, and a small-scale validation experiment.
Next: Evaluation