The Lottery Ticket Was Always There

Here’s a claim that sounds absurd the first time you hear it, and then gets harder to shake the longer you sit with it. Training a neural network might not build anything at all. The trained model may already be present, fully formed, inside the random one you started with, and all that training does is find it.

Start with where this idea came from, because it began as an observation about waste. In 2018 Jonathan Frankle and Michael Carbin noticed something strange about pruning. You can take a large trained network, throw away the overwhelming majority of its connections, keep the ten percent that matter, and the small survivor works about as well as the full network. Fine, networks are overbuilt, we knew that. But then they tried to train that small survivor from scratch, on its own, and it usually failed. Unless, and this was the strange part, you rewound its surviving connections to the exact random values they’d had at the very beginning, before any training. Do that, and the little subnetwork trained up beautifully. They called those lucky subnetworks winning lottery tickets. The full network was a bag of millions of tickets, and training was mostly the process of one lucky ticket winning while the rest were discarded.

That’s the original Lottery Ticket Hypothesis, and it’s already unsettling, because it moves the magic earlier. The capability wasn’t created by training. It was latent in the initial random draw, and training found it.

Then it got stronger. In 2020 Eran Malach and colleagues proved a harder version, one that had been conjectured a year earlier. The Strong Lottery Ticket Hypothesis says you don’t need to train the winning subnetwork at all. A sufficiently large randomly initialized network already contains, somewhere in its untouched random weights, a subnetwork that computes your target function to any accuracy you like, with no training whatsoever. You don’t adjust a single weight. You just find the right subset and delete everything else. The learning is entirely a matter of selection, not adjustment. They proved it for basic networks first, and since then it’s been extended to convolutional networks and to Transformers, which is the part that stops it from being a curiosity about toy models.

Sit with what that implies. If it’s right, then a large enough random network is not a blank slate waiting to be written on. It’s a haystack that already contains every needle you could reasonably want. The famous line about sculpture, that the statue was always inside the marble and the sculptor just removes what isn’t the statue, turns out to be a decent description of what a trained model might be. The competent network was inside the noise. Optimization is the chisel.

This is why the idea belongs to any argument about inevitability. If the solutions are latent in scale, then scale is doing the essential work, not merely helping. A bigger random network contains more subnetworks, so it’s more likely to contain a good one, so making the network bigger makes the needed capability more likely to already be present before you train. Emergence stops looking like a lucky accident and starts looking like a near-certainty you’re buying with parameters. You’re not hoping intelligence shows up. You’re increasing the odds that it was already in the bag.

Now let me argue against the version of this I just made sound inevitable, because the leap from theorem to worldview is where it gets slippery.

The proof is an existence proof, and existence proofs are quieter than they sound. It says a good subnetwork exists inside a large enough random network. It does not say the subnetwork is easy to find, and “large enough” in the theorems can mean far larger than the network you’d actually train. Finding the needle is its own problem, possibly a brutally hard one, and in practice gradient descent doesn’t go hunting for a fixed subnetwork inside frozen random weights. It moves all the weights around. So the tidy picture, real network was hiding in the noise and training just uncovered it, is a story the math permits but doesn’t quite tell. What the math strictly gives you is: the solution’s existence doesn’t require training, only its discovery might.

And there’s a deeper hole. Every version of this result assumes you already know the target function, the thing you’re approximating. It tells you a random network contains a subnetwork that matches a network you already have. That’s a statement about representation, about what large random networks can contain. It says nothing about where the target comes from, or how you’d recognize the right subnetwork without already possessing the answer. The hardest part of intelligence isn’t fitting a known function. It’s figuring out which function is worth fitting. The lottery ticket results are silent on exactly that, which is the part I care about most.

So here’s what I’ll actually claim, narrowed to what survives the objections. The capacity for a huge range of capabilities is genuinely latent in a large network before training, and that’s not a metaphor, it’s proven for the architectures we use. Scale really does make the needed solution more likely to be present. What isn’t settled is whether being present is the hard part or the easy part, whether the story of intelligence is mostly about the haystack containing the needle or mostly about the search that has no map to it.

Which turns the lottery metaphor back on itself in a way I can’t resolve. If every ticket you could want is already in the bag, then the interesting question was never whether you hold a winner. You do. It’s whether you can tell which one it is before the drawing, and nothing in the mathematics promises you can.