Module 03: Transformers

Prerequisites

• Complete Deep Learning.
• Be able to reason about batched matrix multiplication and softmax.

Outcomes

• Derive self-attention, masking, residual, and feed-forward computations.
• Explain tokenization and positional choices as modeling and systems decisions.
• Estimate attention and KV-cache memory.

Ordered free resources

1. Hugging Face LLM Course · Tier A · Course / documentation / labs — begin with tokenization and transformer interfaces.
2. Dive into Deep Learning · Tier A · Book / course / labs — derive attention and transformer blocks.
3. Stanford CS336 — Language Modeling from Scratch · Tier A · Course / notes / assignments — connect architecture details to language-model construction.
4. Neural Networks — Zero to Hero · Tier B · Video / course / code — implement compact autoregressive models and inspect internals.

Checkpoints

• Write every tensor shape for multi-head causal attention.
• Complete the Tiny Attention Lab.
• Estimate KV-cache bytes for two sequence lengths and explain the scaling.

Self-tests

1. Why is the attention mask applied before softmax?
2. Which tensors can be reused during autoregressive decoding?
3. When can a tokenizer change both quality and serving cost?

Capstone

Build a tiny decoder-only transformer and produce a shape trace, causal-mask test, tokenizer comparison, and KV-cache budget. Explain one architectural limitation and one measurement that would expose it.

Next: LLM Training