Module 04: LLM Training

Prerequisites

• Complete Transformers.
• Understand optimizer state, activations, loss curves, and data splits.

Outcomes

• Construct compute, token, and memory budgets before training.
• Explain mixed precision, checkpointing, clipping, scheduling, and data mixtures.
• Design proxy experiments and observability for training failures.

Ordered free resources

1. Stanford CS336 (stanford-cs336) — organize the end-to-end language-model training problem.
2. Full Stack Deep Learning (fsdl) — frame data, experimentation, deployment, and operations together.
3. Deep Learning Systems Course (dlsyscourse) — understand memory and execution constraints.
4. Hugging Face LLM Course (hf-llm-course) — inspect fine-tuning workflows and model interfaces.

Checkpoints

• Estimate parameter, gradient, optimizer, and activation memory separately.
• Define a data-mixture audit with deduplication and contamination checks.
• Diagnose a synthetic unstable run from logs and propose the next experiment.

Self-tests

1. Why can lower numerical precision increase effective throughput?
2. What evidence separates a bad data batch from a bad learning rate?
3. Which assumptions make a small model a valid proxy for a larger run?

Capstone

Write a training plan for a small language model under a fixed compute budget. Include token accounting, data governance, memory mitigations, failure alerts, checkpoints, and an ablation sequence that limits wasted runs.

Next: Inference and Serving