Module 04: LLM Training

Prerequisites

• Complete Transformers.
• Understand optimizer state, activations, loss curves, and data splits.

Outcomes

• Construct compute, token, and memory budgets before training.
• Explain mixed precision, checkpointing, clipping, scheduling, and data mixtures.
• Design proxy experiments and observability for training failures.

Ordered free resources

1. Stanford CS336 — Language Modeling from Scratch · Tier A · Course / notes / assignments — organize the end-to-end language-model training problem.
2. Hugging Face LLM Course · Tier A · Course / documentation / labs — inspect fine-tuning workflows and model interfaces.
3. Full Stack Deep Learning · Tier B · Course / video / labs — frame data, experimentation, deployment, and operations together.
4. Neural Networks — Zero to Hero · Tier B · Video / course / code — study compact language-model training loops before scaling up.

Checkpoints

• Estimate parameter, gradient, optimizer, and activation memory separately.
• Define a data-mixture audit with deduplication and contamination checks.
• Diagnose a synthetic unstable run from logs and propose the next experiment.

Self-tests

1. Why can lower numerical precision increase effective throughput?
2. What evidence separates a bad data batch from a bad learning rate?
3. Which assumptions make a small model a valid proxy for a larger run?

Capstone

Write a training plan for a small language model under a fixed compute budget. Include token accounting, data governance, memory mitigations, failure alerts, checkpoints, and an ablation sequence that limits wasted runs.

Next: Inference and Serving