Distributed All-Reduce Reasoning Lab

Question and hypothesis

What does all-reduce guarantee, and which training conventions are layered above it? Hypothesis: with two ranks holding scalars 1 and 3, sum all-reduce leaves 4 on both ranks; obtaining mean 2 requires explicit division unless the chosen API or training wrapper documents otherwise.

Procedure

1. Launch two local processes using a backend supported by the environment; CPU execution is sufficient for semantic checks.
2. Initialize rank r with scalar 2r + 1.
3. Predict each rank’s tensor after sum all-reduce, then assert both equal 4.
4. Divide by world size and assert both equal 2.
5. Repeat with a short vector to verify elementwise reduction.
6. Add timing around synchronized reductions for increasing tensor sizes. Treat results as local observations, not network benchmarks.
7. In a disposable run with a short timeout, make one rank skip or reorder a collective. Capture the resulting error or timeout, then restore matching order.

Never leave the intentionally mismatched run waiting indefinitely.

Interpretation

All-reduce combines values and distributes the result to every participant. Framework-level distributed training may average gradients through an explicit scale or wrapper convention, so reasoning must separate the collective operation from surrounding optimizer logic. Small-message time is often latency-sensitive; larger messages increasingly expose transfer bandwidth and algorithm/topology effects.

Connections

This lab supports Distributed Systems. Use pytorch-distributed for API behavior and nccl-docs for GPU collective details.

Limitations

A two-process local run does not represent multi-node topology, contention, GPU streams, overlap, numerical reduction order, or fault recovery. Backend behavior and diagnostics differ. Validate production conclusions on the actual interconnect and software stack.

Self-test

1. After sum all-reduce over values [1, 3, 5], what does each of three ranks hold?
2. Why might gradient averaging appear automatic in a training wrapper?
3. What causes a collective-order mismatch to hang or fail?
4. Which tensor-size sweep would help reveal the transition from latency to bandwidth sensitivity?