The Economics of AI Training Data

I let AI crawlers read this site. There’s an llms.txt file at the root, on purpose. That small decision is the doorway into a much bigger question: what happens to the economics of data when data becomes the primary asset of the most valuable companies on earth?

The current model is scrape everything and train once. It worked because the web was treated as a commons, and because no individual page was worth anything on its own. Both assumptions are cracking at the same time. Publishers are signing licensing deals or filing lawsuits, sometimes both in the same quarter. The open web is closing in response to exactly this dynamic: paywalls, login walls, robots.txt files that read like cease-and-desist letters. And the model builders are discovering that the marginal value of another billion scraped pages is falling while the value of curated, high-quality, verified data climbs. The scrape-everything era looks less like a permanent state and more like a land rush before the fences went up.

The music industry is the parallel I keep reaching for, because I watched a version of it from inside audio at Remixd. Music went from ownership to streaming, and the economics inverted: the asset stopped being the recording and became the catalog plus the pipe. Data looks like it’s on the same path. Individual creators get fractions of pennies while aggregators capture the value, and the compensation question gets answered by whoever has negotiating power, not by any principle of fairness. A major publisher can sign a licensing deal. A blogger cannot. If your data matters at scale, you license it. If it doesn’t, you get scraped. Most of us are in the second group, which is why “data ownership” as a slogan mostly benefits people who already own a lot of data.

Now the honest counterpoint, which happens to be my own position, so I can’t dodge it. Maybe none of this matters at the individual level. My site’s contribution to any model is statistically indistinguishable from zero, and holding it back accomplishes nothing except making the site less findable in a world where discovery increasingly happens through models. That’s the actual reasoning behind my llms.txt, not generosity. Being in the training data is the new being in the index. When someone asks a model about CTV advertising or solo development and the answer carries a trace of something I wrote, that’s distribution. The 2005 version of me made the same bet with Google, and it paid for twenty years.

Whether the trade stays rational is the part I can’t answer. The Google deal had a currency: traffic. You gave the index your content and the index sent you readers. The model deal pays in something vaguer, presence, influence on the answer, and the models are getting better at answering without sending anyone anywhere. It’s possible I’m giving away the content and the currency this time is nothing. It’s also possible that opting out just makes you invisible while changing nothing else, which is a worse deal still.

The web made a deal once before, content for traffic, and it held for twenty years until it didn’t. The new deal, content for presence in the model, is being written right now, by parties with wildly unequal power, and most of the people supplying the content aren’t at the table. I’ve chosen my side of the bet with a text file at the root of this site. Ask me in ten years whether it was the right one.