Everyone Keeps Reinventing the Same Machine

In evolutionary biology there’s a phenomenon that should be stranger than it feels: the eye evolved independently dozens of times. Camera eyes in vertebrates and, separately, in octopuses. Compound eyes in insects. Different lineages, different starting points, no shared ancestor with the finished organ, and they all converged on the same handful of optical solutions, because physics only allows so many ways to focus light onto a detector. When the problem has a small number of good answers, unrelated searchers keep finding the same ones. Convergence isn’t a coincidence. It’s evidence that the answers were sitting there in the structure of the problem, waiting to be found by anyone who looked hard enough.

I think the same thing is happening in neural network architecture, and it’s the third leg of the case that intelligence is discovered rather than invented.

For most of the last decade the Transformer ruled by itself. Attention, the mechanism where every token looks at every other token, was the thing that made large language models work, and its one great flaw was cost: comparing everything to everything means the work grows with the square of the sequence length, so long inputs get expensive fast. So people went hunting for alternatives that would scale better, and they came at it from genuinely different directions. One camp revived state-space models, an idea borrowed from control theory and signal processing, machinery for describing how a system’s hidden state evolves over time. That line produced Mamba. Another camp came from the world of recurrent networks, the old sequential architectures everyone had supposedly abandoned when attention arrived, and tried to rebuild them so they could train in parallel. That line produced RWKV. Control theory and recurrent networks are not the same neighborhood. These were different intuitions, different math, different people.

And then in 2024 Tri Dao and Albert Gu proved the punchline, in a paper whose title is the whole argument: “Transformers are SSMs.” They showed that a Transformer’s attention and a state-space model like Mamba are two faces of a single underlying operation, connected through a class of structured matrices, so that the same computation can be written either as attention or as a state-space recurrence. Not similar. Dual, two views of the same object, the way the same rotation can be described with matrices or with quaternions. RWKV, arriving from the recurrent side, lands in the same territory, close enough that the whole family, attention and state-space models and modern linear recurrences, is now understood as variations on one theme rather than rivals.

That’s the octopus eye again. Separate teams, separate starting intuitions, converging on architectures that turn out to be mathematically the same machine. And the reading that fits the inevitability thesis is the one I find hard to resist: they’re not each inventing something out of nothing. They’re circling an attractor in the space of possible architectures, a small set of good answers that the problem itself defines, and their different routes are different approaches to the same destination. If that’s true, then architecture research isn’t really invention. It’s cartography. The good designs exist independently of us, the way the camera eye existed as a possibility long before any animal found it, and we keep rediscovering them because there aren’t many places to land.

Here’s where I have to argue with myself, because this is the pillar I’m least sure of, and the convergence story has a way of feeling more profound than it is.

The most deflating objection is that the convergence is manufactured, not discovered. All these architectures are being trained on the same benchmarks, optimized for the same hardware, published in the same venues, built by people who read each other’s papers. GPUs are exquisitely good at large matrix multiplications and mediocre at almost everything else, so of course every successful architecture reduces to stacks of matrix multiplications. That’s not the universe revealing its deep structure. That’s a thousand researchers all pointed at the same NVIDIA chip and the same leaderboard, filing down their ideas until they fit. Convergence under identical selection pressure isn’t spooky. It’s what selection pressure does. Change the hardware and the “attractor” might move, which is not how a law of nature behaves.

And there’s a subtler problem. Proving two architectures are mathematically dual is a statement about a small, clean family of models, the ones simple enough to prove things about. The actual frontier systems are messy, full of components bolted on because they worked, and nobody has shown that the whole messy apparatus of a real large model converges to anything. The convergence lives partly in the theory papers, where the objects are tidy enough to relate, and the messy empirical reality may be more contingent than the clean duality suggests. I’d be overreaching if I let a theorem about simplified models carry a claim about every system in production.

So here’s the honest residue. The mathematical duality is real, proven, and genuinely surprising, and it does show that ideas from far-apart fields collapse onto shared structure, which is at least a hint that the structure is real and not arbitrary. What I can’t cleanly separate is how much of the convergence is the problem’s deep geometry showing through and how much is just everyone optimizing against the same chip and the same test. Both are operating. I don’t know the mix, and anyone who tells you they do is reading more into a leaderboard than it can hold.

Which leaves the question the eye already asked, transposed into silicon. If independent searchers keep finding the same architectures, the interesting thing was never that they agree. It’s whether they’re converging because the answers are woven into the problem, the way optics constrains eyes, or because they’re all squinting at the same benchmark under the same lights. From inside the search, standing at the leaderboard, those two look identical, and I don’t yet know how you’d tell them apart.