Module 03: Transformers

Prerequisites

• Complete Deep Learning.
• Be able to reason about batched matrix multiplication and softmax.

Outcomes

• Derive self-attention, masking, residual, and feed-forward computations.
• Explain tokenization and positional choices as modeling and systems decisions.
• Estimate attention and KV-cache memory.

Ordered free resources

1. Hugging Face LLM Course (hf-llm-course) — begin with tokenization and transformer interfaces.
2. Dive into Deep Learning (d2l) — derive attention and transformer blocks.
3. Stanford CS336 (stanford-cs336) — connect architecture details to language-model construction.
4. Zero to Hero (karpathy-zero-to-hero) — implement compact autoregressive models and inspect internals.

Checkpoints

• Write every tensor shape for multi-head causal attention.
• Complete the Tiny Attention Lab.
• Estimate KV-cache bytes for two sequence lengths and explain the scaling.

Self-tests

1. Why is the attention mask applied before softmax?
2. Which tensors can be reused during autoregressive decoding?
3. When can a tokenizer change both quality and serving cost?

Capstone

Build a tiny decoder-only transformer and produce a shape trace, causal-mask test, tokenizer comparison, and KV-cache budget. Explain one architectural limitation and one measurement that would expose it.

Next: LLM Training