Why does random weight noise work on big models?

Question

Teach me this page — https://arxiv.org/abs/2603.12228

askNib · Accepted Answer

Normally we finetune a huge model with careful gradient steps. This paper asks a weirder question: what if you just jiggled the weights randomly instead? Picture the pretrained weights as one point in a huge space. Around that point is a whole cloud of nearby parameter settings. In a small model, task experts hide in a tiny sliver of that cloud — like needles in a haystack. You need gradient descent to find them. But in a large, well-pretrained model, that neighborhood is packed — task-solving experts are everywhere nearby. Random noise practically trips over one. That density claim gives a wild recipe: sample N random perturbations of the weights, no gradients, no backprop at all. Test each one on the task, keep only the top K performers — the accidental experts that random luck happened to land near. Then don't pick a winner — run all K experts on new inputs and let them vote. Diversity plus majority vote beats any single noisy model. The surprise: this dumb, fully parallel, gradient-free method is competitive with PPO, GRPO, and evolution strategies on today's large models. So the real insight isn't the algorithm — it's the geometry. Pretraining at scale doesn't give you one good point, it gives you a thicket of good points. Scale changes the shape of the search itself — with a big enough pretrained model, you barely need to search at all, just to look around.