The machines are starting to talk to themselves — and no one’s sure AI model collapse hasn’t already begun.
What started as a quiet murmur in research papers and conference halls is now gaining volume. Could artificial intelligence, the very force driving modern tech’s wild acceleration, be training itself into confusion?
It’s not science fiction. The risk of AI model collapse is very real. And the way it unfolds is stranger — and quieter — than most people expect.
No crash, no alarms. Just a slow drift, like a sharp photograph softening with each new copy.
The alarm rang loudest in July 2024, with a Nature paper from Oxford’s Ilia Shumailov and his colleagues. “Indiscriminate use of model-generated content in training causes irreversible defects in the resulting models,” they wrote.
Shumailov’s team described AI model collapse as “a degenerative process affecting generations of learned generative models, in which the data they generate ends up polluting the training set of the next generation.”
Their experiments revealed how recursive training — models feeding on their own outputs — starts with small losses and leads to major distortions. “Over time,” they warned, “models start losing information about the true distribution, which first starts with tails disappearing.”
It wasn’t just academic. In one striking experiment, a language model began inserting the word “jackrabbits” into completely unrelated text — a sign that the system’s connection to its original knowledge was breaking down.
Their findings painted a stark picture of recursive training, a process that can be likened to making a copy of a copy of a copy. As Shumailov and team detailed in their paper, each generation gets further from reality.
And when that reality is blurred, the risks spread. The compounding errors are akin to a high-stakes, statistical version of the children’s game ‘telephone,’ where each iteration risks distorting the original message further — only here, the ‘whispers’ are complex data transformations, and the potential for misinterpretation is amplified by mathematical processes.
So far in 2025, the big question is whether AI model collapse has started creeping into real-world systems.
There’s no massive failure yet. But in April, researchers from Bloomberg flagged risky behavior in retrieval-augmented generation (RAG) models — AIs that blend internal training with outside information sources. Even with that added information, some outputs were still off.
“This has far-reaching implications given how ubiquitously RAG is used,” said Amanda Stent, Bloomberg’s head of AI research and strategy, during the AI at Scale Forum. “Recursive self-training is not theoretical — it’s operational.”
Is that outright AI model collapse? Not quite — but drift is harder to dismiss now.
One researcher summed up the anxiety during a closed session at MIT: “It’s not the flood we’re afraid of — it’s the mold.”
Fortunately, researchers aren’t sitting still. Shumailov’s team argued that access to original, varied datasets is crucial—especially for fields where rare, hard-to-find knowledge matters most. Think of it like a seed vault for language and culture, saving original, human-made content before it gets lost in a sea of algorithmic echoes.
Another fix gaining traction is data provenance—clear labeling of whether something was made by humans or machines. That could help future AIs avoid the recursion trap entirely.
Some believe today’s biggest AI models already enjoy a kind of first-mover advantage: they were trained on the web before it filled up with machine-made content. But that advantage won’t last forever unless developers build stronger guardrails and better ways to test what’s real, not just what sounds good.
No one’s calling it a crisis yet — but it’s closer to the surface now than it was a year ago. And the longer the AI world waits, the harder it may be to untangle the mess later.
It’s a strange thought: machines stuck in endless conversations with their own reflections, forgetting how to speak to the humans they were built to serve.
We’re not there yet — but the outlines of AI model collapse are already visible, like a shoreline appearing out of fog.
Right now, AI model collapse is still a warning, not a wreckage. But if machines forget how to learn from real life — and start muttering only to themselves — we might not recognize what’s missing until it’s already gone.
