Module 08: Evaluation

Prerequisites

• Complete the first seven track modules.
• Be able to reproduce a run, define an SLO, and inspect data provenance.

Outcomes

• Turn product and research claims into measurable evaluation questions.
• Combine capability, human, safety, reliability, latency, throughput, and cost evidence.
• Report uncertainty, contamination risk, limitations, and regressions honestly.

Ordered free resources

1. Language Model Evaluation Harness (lm-eval-harness) — inspect reproducible task and metric execution.
2. HELM (helm) — study multi-metric, scenario-based evaluation.
3. Full Stack Deep Learning (fsdl) — connect offline evaluation to production monitoring.
4. Hugging Face LLM Course (hf-llm-course) — revisit task pipelines and model comparisons.

Checkpoints

• Write an evaluation contract: claim, population, data, metric, threshold, and decision.
• Run one paired model comparison with fixed prompts and decoding settings.
• Add latency, cost, and failure-rate reporting to a quality benchmark.

Self-tests

1. When does a higher average score fail to support a deployment decision?
2. How can decoding settings make a model comparison unfair?
3. Which evidence would reveal benchmark contamination?
4. Why must regressions be sliced by scenario rather than only aggregated?

Capstone

Complete the track capstone from Full-Stack LLM Systems with a reproducible evaluation report. Include decision thresholds, paired comparisons, system metrics, failure slices, uncertainty, contamination checks, and explicit limits on generalization.