Why do fast chips sit idle?

1the puzzle

Here's a puzzle: a chip that can do almost a quadrillion operations every second often sits mostly idle. Why would the world's fastest hardware be waiting around?

2two clocks

Every computation has two clocks. Math time is FLOPs divided by chip speed. Comms time is bytes moved divided by bandwidth. The slower clock decides how long you wait.

3intensity

Divide FLOPs by bytes and you get arithmetic intensity, FLOPs per byte. Compare it to the chip's own ratio. On a TPU v5e, that's 1.97e14 over 8.2e11, about 240.

4the roofline

This is the roofline. Below 240 FLOPs per byte, performance climbs with intensity, capped by memory bandwidth. Past it, you hit the flat roof: full compute speed.

5dot product

A dot product shows the trap. Each element is loaded, used once, and thrown away: about half a FLOP per byte. Far below 240, so it can never use the chip.

6the matmul

Now the matmul, the star of deep learning. Multiply a B by D batch against a D by F weight matrix. The FLOPs grow as B times D times F, but the bytes are mostly the weights.

7the 240 rule

So the rule is startlingly simple: on a TPU, you become compute-bound once your batch exceeds about 240 tokens. GPUs land near 300. Below that, you're paying for chip you can't use.

8between chips

The twist: it isn't only memory. Split a matmul across two chips and the partial sums must cross the network, at just 4.5e10 bytes per second. That wire is the new roof.

9D, not B

Here's the surprise. The threshold now depends on F, not B. Compute scales with D, but the bytes sent scale with B and F, so D cancels; you need D above about 8755 to stay compute-bound.

10the whole idea

So the keepsake: every operation is a race between math and bytes. Intensity decides who wins, hardware sets the line, and batch size, about 240 tokens, is your lever.

11why not size

Great catch. Bigger matrices do more math, but they also move more bytes, in step. Look at the ratio: FLOPs are two B D F, and the weight bytes are two D F.

12reuse

Picture it: double the matrix and you double the FLOPs and double the weight bytes. Nothing changes. But reuse each loaded weight across more batch rows, and math rises while bytes stay put.

13sharded matmul

Now the network. Shard the contraction dimension D across two chips. Each holds half of X and half of Y, and computes only a partial sum of the output.

14the exchange

To add them, the chips swap those partial outputs over the network, two bytes per entry. Cost is bytes over bandwidth: B times F, over the slow wire at 4.5e10.

15D takes over

Compare the clocks. B and F sit in both, so they cancel, leaving D. Now matrix size does matter: only a big contraction dimension, above about 8755, hides the network behind math.