Module 06: GPU Kernels and Compilers

Prerequisites

• Complete Inference and Serving.
• Be able to estimate bytes moved, operations performed, and tensor layouts.

Outcomes

• Explain GPU hierarchy, warps/wavefronts, memory spaces, occupancy, and synchronization.
• Use profiling evidence to distinguish bandwidth, latency, and compute limits.
• Describe how kernel DSLs and compilers transform tensor programs.

Ordered free resources

1. CUDA C++ Programming Guide (cuda-guide) — build the vendor-specific execution and memory model.
2. AMD HIP Documentation (amd-hip) — compare portable concepts and hardware terminology.
3. Triton Tutorials (triton-tutorials) — express tiled kernels with explicit program instances.
4. Apache TVM Documentation (apache-tvm) — study scheduling, lowering, and generated code.
5. Deep Learning Systems Course (dlsyscourse) — reconnect kernels to framework execution.

Checkpoints

• Draw and explain the hierarchy in CUDA Execution Concepts.
• Predict coalescing and reuse for two matrix layouts.
• Compare a simple reference operation with one fused or tiled implementation.

Self-tests

1. Why can high occupancy fail to imply high performance?
2. Which synchronization scope is valid inside one thread block?
3. What evidence shows that fusion reduced total memory traffic?

Capstone

Optimize one small GPU operation using either CUDA/HIP, Triton, or a compiler schedule. Record the baseline, bottleneck hypothesis, correctness checks, profile evidence, speedup boundaries, and portability limitations.

Next: Distributed Systems