Module 05: Inference and Serving

Prerequisites

• Complete LLM Training checkpoints.
• Understand causal decoding, precision, and KV-cache scaling.

Outcomes

• Decompose request latency into queueing, prefill, decode, and transfer.
• Explain batching, scheduling, cache allocation, quantization, and parallel serving.
• Define capacity tests and SLOs that include quality constraints.

Ordered free resources

1. vLLM Documentation · Tier A · Documentation / tutorials / code — study continuous batching and cache-aware serving concepts.
2. NVIDIA Triton Inference Server Documentation · Tier A · Documentation / tutorials / code — inspect production model-serving controls.
3. Deep Learning Systems · Tier A · Course / video / labs — connect serving behavior to framework and hardware execution.
4. Triton Language Tutorials · Tier A · Documentation / tutorials / labs — understand the kernels behind optimized inference paths.

Checkpoints

• Build a latency budget for short-prompt and long-prompt workloads.
• Calculate concurrency limits from a KV-cache memory budget.
• Design a load test that reports percentiles, throughput, errors, and output quality.

Self-tests

1. Why can higher throughput worsen user-visible latency?
2. When does prefill dominate, and when does decode dominate?
3. Which quality checks must accompany a quantization benchmark?

Capstone

Create a serving design for two workload classes with different SLOs. Specify admission control, batching, cache policy, fallback behavior, telemetry, and a benchmark plan that prevents averages from hiding tail failures.

Next: GPU Kernels and Compilers