Module 05: Inference and Serving

Prerequisites

• Complete LLM Training checkpoints.
• Understand causal decoding, precision, and KV-cache scaling.

Outcomes

• Decompose request latency into queueing, prefill, decode, and transfer.
• Explain batching, scheduling, cache allocation, quantization, and parallel serving.
• Define capacity tests and SLOs that include quality constraints.

Ordered free resources

1. vLLM Documentation (vllm-docs) — study continuous batching and cache-aware serving concepts.
2. NVIDIA Triton Inference Server (triton-inference-server) — inspect production model-serving controls.
3. HELM (helm) — connect deployment choices to scenario-level quality and efficiency.
4. PyTorch Tutorials (pytorch-tutorials) — prototype profiling, compilation, and inference behavior.

Checkpoints

• Build a latency budget for short-prompt and long-prompt workloads.
• Calculate concurrency limits from a KV-cache memory budget.
• Design a load test that reports percentiles, throughput, errors, and output quality.

Self-tests

1. Why can higher throughput worsen user-visible latency?
2. When does prefill dominate, and when does decode dominate?
3. Which quality checks must accompany a quantization benchmark?

Capstone

Create a serving design for two workload classes with different SLOs. Specify admission control, batching, cache policy, fallback behavior, telemetry, and a benchmark plan that prevents averages from hiding tail failures.

Next: GPU Kernels and Compilers