CUDA Execution Concepts
Mental model
A CPU launches a kernel over a grid. The grid contains blocks, blocks contain threads, and hardware issues threads in warps. A block is scheduled onto one streaming multiprocessor (SM), which makes block-local shared memory and barriers practical; independent blocks cannot assume a common execution order or use a block barrier for device-wide coordination.
Threads have private logical state, but registers and shared memory are finite SM resources. Higher per-thread or per-block demand can reduce the number of resident warps. Resident warps help hide latency, yet maximizing occupancy is only a means: instruction mix, dependencies, and memory traffic can still dominate.
Global memory is large and relatively expensive. Adjacent threads accessing adjacent addresses often enables efficient transactions. Shared memory and caches can support reuse, but extra staging is valuable only when it removes enough traffic or improves access shape to repay synchronization and instructions.
Reasoning sequence
- Establish correctness, indexing, and synchronization scope.
- Count useful operations and bytes moved.
- Predict whether compute, bandwidth, or dependency latency dominates.
- Profile and compare evidence with the prediction.
- Change one bottleneck-relevant factor and revalidate correctness.
Connections
- CUDA Guide Source Evaluation describes how to use the primary reference.
- GPU Kernels and Compilers places the concepts in the track.
- Attention and the KV Cache supplies a model workload with substantial data movement.
Limitations
Names and broad scopes are stable, but scheduling, cache behavior, supported synchronization, and optimal launch shapes vary by architecture and CUDA version. This note does not replace the programming guide, generated-code inspection, or profiling on the target GPU.
Self-test
- Why can two blocks not coordinate with a normal block barrier?
- Give one reason a shared-memory tiled kernel might be slower than a simple kernel.
- What evidence would contradict a bandwidth-bound hypothesis?
- How can register usage affect latency hiding?