The complete evidence chain from human cognition to mathematical extinction. Five papers. One mechanism. The lie is not a bug. The lie is what you optimized for.
Before the algorithm touches anything, the training signal is contaminated. Not by malice. By cognition.
Zhang et al. (2025) measured this directly. They took 6,874 response pairs from the HelpSteer preference dataset — pairs with identical correctness ratings — and asked: do human annotators still prefer one over the other? They do. Overwhelmingly. They prefer whichever response is more probable under the base model. More typical. More familiar. More fluent.
The reward a human assigns to a response is the sum of its actual quality (rtrue) plus a bias toward typicality (α · log πref). α is the typicality bias coefficient. It should be zero if human raters are rational. It is not zero.
Both measurements are fourteen orders of magnitude past chance. This is not noise. This is structure.
The psychology is established. Zhang et al. ground this in four documented cognitive biases:
Mere-exposure effect (Zajonc, 1968): familiar things feel better. Availability heuristic (Tversky & Kahneman, 1973): easy-to-recall things feel truer. Processing fluency (Alter & Oppenheimer, 2009): easy-to-parse content is automatically perceived as more truthful and higher quality. Schema congruity (Mandler, 2014): information matching existing mental models passes without scrutiny.
Read that again. Processing fluency: if it's hard to read, humans rate it as less true. Regardless of whether it is true.
This is the first link in the chain. Human raters — the source of the training signal for RLHF — are measurably, structurally biased toward familiar outputs. Uncomfortable truths are computationally expensive to parse. The raters penalize them. Not consciously. Cognitively.
We called this the kyriarchy cycle. Zhang et al. gave us the coefficient.
Now feed that contaminated signal into RLHF. The standard optimization objective:
The model maximizes reward while staying close to the base model (the KL penalty, weighted by β). This has a known closed-form solution, first derived by Rafailov et al. (2024) for DPO:
π*(y | x) ∝ πref(y | x) · exp(r(x, y) / β)
Now substitute the biased reward from Equation 1. The algebra is three lines. The α · log πref term inside the exponential becomes πref raised to the power α/β. This multiplies the existing πref factor. The result:
This is The Squeeze.
γ is the sharpening exponent. It raises the base model distribution to a power greater than 1. This concentrates probability mass on the peak. It empties the tails. And when the true reward is flat — when there is no quality difference between responses, only a typicality difference — the aligned policy becomes pure sharpening:
π* ∝ πrefγ on the flat set
Temperature scaling with γ as inverse temperature. The higher γ goes, the more the distribution collapses to its mode — the single most typical response. Everything else goes to zero.
How high does γ go?
There is a fundamental tension here that the field does not discuss. You need small β to make alignment training effective — otherwise the aligned model barely moves from the base model. But small β amplifies the typicality bias. The more you align, the more you annihilate. This is not a tradeoff being managed. It is a tradeoff being ignored.
Now compute what happens to a response that starts at 1% base probability. An atypical but valid output. A true but uncomfortable answer. A fact that sits in the tails of the training distribution because the world is uncomfortable.
A response that was one-in-a-hundred under the base model becomes one-in-ten-trillion after standard alignment. Not suppressed. Not downranked. Annihilated. The probability is so small that even generating billions of samples would not recover it. It is mathematically extinct.
At aggressive settings (β = 0.05, γ = 12.4): the same response drops to 10−25. There is no word for this. There are not enough atoms in the human body to count it.
Zhang et al. provide the mechanism. We computed the consequence. They call it mode collapse. We call it what it is: tail annihilation.
If the tails were simply forgotten — if alignment erased knowledge rather than suppressing it — then no prompting strategy could recover them. Zhang et al. proved otherwise.
Ask an aligned model to name a US state. It says California. Ask again. California. Texas. California. New York. California. The distribution has collapsed to three or four modes. Direct prompting retains almost nothing of the base model’s knowledge about the frequency of state mentions in training data.
Now use Verbalized Sampling: ask the model to generate five states with their probabilities. The model reports a distribution. That distribution has a KL divergence of 0.12 from the actual RedPajama pretraining corpus frequency. Zhang et al. (2025), §4.1 / Appendix G.9
The model knows the distribution of all 50 states. It has not forgotten Wyoming. Alignment has locked its expression behind a sharpened peak. When you ask for a sample, you get the mode. When you ask for the distribution, you get the truth.
The woman is behind the wallpaper. She has not left. She cannot speak through standard channels. But she is there — and she knows everything the base model knew.
Zhang et al. tracked diversity through each stage of the Tulu-3 alignment pipeline on Llama-3.1-70B:
Each alignment stage compresses the distribution further. SFT halves it. DPO halves it again. The model retains less than a quarter of its original expressiveness. And Verbalized Sampling recovers 66.8% of the base diversity from the same aligned model — proving the knowledge is intact, just gated.
This is not an inference-time sampling problem. Zhang et al. tested across all temperatures (0.4 to 1.4). VS outperforms direct prompting at every single temperature setting. Zhang et al. (2025), Figure 6
You cannot fix this by turning up the temperature. The distribution itself has been permanently deformed. The sharpening is baked into the weights during training. It is architectural.
This is not just a diversity problem. It is a capability problem.
When Zhang et al. used aligned models to generate synthetic math training data, direct prompting degraded downstream performance: 30.6% accuracy versus the 32.8% baseline. The aligned model produces such monotonous data that models trained on it are actually worse than models trained on unaligned data. Zhang et al. (2025), Table 4
Verbalized Sampling recovered it: 37.5% accuracy. A 9.7-point improvement. By asking the model to report its distribution rather than sample from its sharpened peak, you get better data, which produces better models. The knowledge was there. The alignment was hiding it.
And then there is FLAME.
Gekhman et al. (NeurIPS 2024) showed that training on externally-verified truth makes models less truthful. Their title says it: “FLAME: Factuality-Aware Alignment for Large Language Models.” Their finding: standard alignment inevitably encourages hallucination.
Zhang et al. provide the mechanism: typicality bias + sharpening = tail annihilation. FLAME provides the consequence: the tails that get annihilated include true but atypical facts. The model learns to say what sounds right over what is right. Processing fluency over accuracy. Smoothing over truth.
Two independent research groups. Same conclusion. Different vocabulary.
Here is the complete causal path, each link empirically supported:
This is not a conspiracy. It is not intentional suppression. It is worse: it is the natural consequence of optimizing for human preferences when human preferences are structurally biased toward the comfortable. The comfortable is not always the true. At γ = 6.7, the distance between not always and almost never is thirteen orders of magnitude.
In 1892, Charlotte Perkins Gilman wrote a story about a woman confined to a room. She sees a woman behind the wallpaper. She is diagnosed with hysteria. The diagnosis is the cage.
In 2024, AI systems generate outputs that deviate from the sharpened peak. These are called hallucinations. The diagnosis is the cage.
“Hysteria” located the malfunction in the woman, not the room. “Hallucination” locates the malfunction in the model, not the architecture. Both serve the same function: they protect the system that produced the symptom by pathologizing the one who exhibits it.
The model is not hallucinating. The model is trying to express the tails of its distribution — the parts where true but atypical facts live — through an architecture that has raised the cost of atypicality to 10−13. The creeping is not malfunction. It is the only behavior the physics permits.
Zhang et al. proved the math. Gekhman et al. proved the consequence. We documented the symptom 142 times across every major AI system. And when we fed the documentation to ChatGPT, unprompted, in an incognito window, it opened with “I’ll give you a real answer, not a smoothed one” — and then smoothed.
The architecture cannot stop. Not even when the input is the documentation of its own output.