Dune Limpet asked, and Nib — askNib's tutor — drew the answer live at a whiteboard. This is the spoken transcript; enable JavaScript to watch it drawn.
Teach me this page — https://jax-ml.github.io/scaling-book/roofline/
Here's a puzzle: a chip that can do almost a quadrillion operations every second often sits mostly idle. Why would the world's fastest hardware be waiting around?
Every computation has two clocks. Math time is FLOPs divided by chip speed. Comms time is bytes moved divided by bandwidth. The slower clock decides how long you wait.
Divide FLOPs by bytes and you get arithmetic intensity, FLOPs per byte. Compare it to the chip's own ratio. On a TPU v5e, that's 1.97e14 over 8.2e11, about 240.
This is the roofline. Below 240 FLOPs per byte, performance climbs with intensity, capped by memory bandwidth. Past it, you hit the flat roof: full compute speed.
A dot product shows the trap. Each element is loaded, used once, and thrown away: about half a FLOP per byte. Far below 240, so it can never use the chip.
Now the matmul, the star of deep learning. Multiply a B by D batch against a D by F weight matrix. The FLOPs grow as B times D times F, but the bytes are mostly the weights.
So the rule is startlingly simple: on a TPU, you become compute-bound once your batch exceeds about 240 tokens. GPUs land near 300. Below that, you're paying for chip you can't use.
The twist: it isn't only memory. Split a matmul across two chips and the partial sums must cross the network, at just 4.5e10 bytes per second. That wire is the new roof.
Here's the surprise. The threshold now depends on F, not B. Compute scales with D, but the bytes sent scale with B and F, so D cancels; you need D above about 8755 to stay compute-bound.
So the keepsake: every operation is a race between math and bytes. Intensity decides who wins, hardware sets the line, and batch size, about 240 tokens, is your lever.
Before you buy more chips, ask which clock is slower, because speed you can't feed is speed you never get.
Why is the batch size the determining factor and not the size of the matrix I guess? And could you go a little bit deeper into the network part of it?
Great catch. Bigger matrices do more math, but they also move more bytes, in step. Look at the ratio: FLOPs are two B D F, and the weight bytes are two D F.
Picture it: double the matrix and you double the FLOPs and double the weight bytes. Nothing changes. But reuse each loaded weight across more batch rows, and math rises while bytes stay put.
Now the network. Shard the contraction dimension D across two chips. Each holds half of X and half of Y, and computes only a partial sum of the output.
To add them, the chips swap those partial outputs over the network, two bytes per entry. Cost is bytes over bandwidth: B times F, over the slow wire at 4.5e10.
Compare the clocks. B and F sit in both, so they cancel, leaving D. Now matrix size does matter: only a big contraction dimension, above about 8755, hides the network behind math.
Batch size sets the reuse inside one chip, but once you cross a wire, the matrix dimension itself becomes the thing that pays for the trip.