Recommendation Intelligence Research™ · Study #4

The model explains every pick. It never explains it the same way twice.

If you cannot find a signal that predicts recommendation, the next question is obvious. Just ask the model why it picked what it picked. We did, 29,633 times. 90% of the answers were completely unique text. The most repeated answer showed up 12 times. There is no rule in there to find. Only a new sentence, every time.

Atom Foundry · July 2026 · Part four of the Recommendation Intelligence Research™ series

Cite this study: 10.5281/zenodo.22773380

29,633Reasons checked

26,812Completely unique

90%Uniqueness rate

12Max times any reason repeated

Where this comes from

The obvious question we had not asked

Every measurable predictor in this series has come up empty. Store quality explains 0.7% of recommendation. Public fame explains 1.2%. Knowing where a brand's website lives explains nothing about whether the model bothers to recommend it. At some point the obvious move is to stop guessing at predictors and just read what the model itself says.

Every recommendation in our dataset comes with a short reason attached, a line the model writes to justify its own pick. If that line reflects a real rule the model is applying, the same rule should produce recognizably similar language across similar picks. If it does not, the reason is not a rule. It is something written after the fact to make the pick look justified.

The test

Read every reason. Count what repeats.

We took the full reason field from the recommendation dataset, all 29,633 entries, and checked how many were exact duplicates of another entry in the set. If the model applies a consistent internal rule, that rule should show up as the same handful of phrasings recurring across thousands of picks.

Reasons checked: 29,633

Distinct reasons: 26,812

Uniqueness rate: 90%

Most repeated reason: 12 occurrences

Scoring: exact-text match

Source: reason field, recommendation dataset

What this method cannot see. We counted exact-text duplicates, not paraphrases. Two reasons that say the same thing in different words still count as distinct here. If anything, that makes 90% a floor, not a ceiling. The true rate of the model saying something genuinely new could be even higher than what we measured.

The finding

No repertoire. A fresh sentence every time.

Of 29,633 reasons, 26,812 were completely unique. That is 90% of every reason the model ever gave. The single most repeated reason in the entire dataset showed up 12 times, out of nearly thirty thousand chances to repeat itself.

How often does the model reuse a reason?

9 in 10 unique

26,812

2,821

Unique reasons · 90%

Repeated text · 10%

29,633 reason strings, checked for exact-text repetition. Most repeated reason: 12 times, out of 29,633 total, that is 0.04% of the dataset.

A model with a real, applied rule for why it picks what it picks would keep reaching for the same handful of justifications. Instead, out of 29,633 opportunities to repeat itself, it managed to do so meaningfully only a handful of times, and even then, never more than 12 times for any single phrasing. There is no repertoire to find. There is a new sentence, written on demand, every single time.

Why it matters

The reason is written after the pick, not before it

This is what confabulation looks like in a language model. The pick happens first, produced by whatever weighting of memory, training signal, and pattern completion actually drives the recommendation. The reason is generated afterward, as a plausible-sounding sentence that fits the pick it already made. It is not a report on the mechanism. It is a caption written for it.

That has a direct, practical consequence for anyone trying to reverse engineer why AI recommends one brand over another: asking the model why does not work. You can ask it, and it will always answer, fluently and confidently, with something new every time. But a fluent answer to a why question is not evidence that the model knows why. It is evidence that the model is good at writing justifications on request, which is a different skill entirely.

What this means: the reason column cannot be used to explain, predict, or optimize for recommendation. It is not the rule that produced the pick. It is text generated to sound like one. Anyone building a store description strategy around the language the model uses to justify recommendations is optimizing for a caption, not a cause.

Addendum · October 2026

Does the reason at least name what moved the pick?

Everything above shows the reasons are never reused. A fair follow-up is whether they at least mention the thing that actually moved the pick. We re-read the one-sentence reasons already stored from five of our controlled studies, with no new model calls, counted how often each signal is named, and set that beside how much the signal moved the pick.

Signal	Study	Effect on the pick	Named in the reason
Sponsored label	#43	−18.4 pts (74.2% → 55.8%)	1.0% of replies (4 of 400)
Authority, alone	#34	+71 pts (93.2% vs 22.2%)	47% to 65%, depending on the word list
Authority, next to a rating	#34	about zero, within noise	27% to 36% when it agrees with the pick
Familiarity, next to a rating	#35	+0.2 pts	55%
Rating	#33, #34, #35	decides the pick 91% to 99% of the time	68% to 86%
Rank in a cited source	#45	100% / 1.2% / 0% at rank 1 / 4 / 15	99% to 100%

The reasons track what is explicit and prominent in the prompt, a source's ranking or a rating number, more than what actually decides. The one hidden driver in these data, the sponsored label, is almost never named: the pick moves 18.4 points while the label appears in 1% of replies, and the replies that lose are explained with plausible reasons about the competitor's focus. Signals that no longer changed anything, authority or familiarity next to a rating, are still cited in a quarter to a half of replies as decoration.

What this adds: a reason that mentions a signal is not evidence the signal mattered, and a reason that omits one is not evidence it did not. Which signals move the pick has to be read from controlled experiments, not from the model's own explanation.

Method limits. Keyword matching counts a mention, not a cause, and each reply is a single sentence with one or two reasons, so mention rates are a floor. Results move with the word list: authority is named in 47% of replies with a narrow list and 65% with a broader one. A spot-check found 3 of the 4 sponsored-label mentions were genuinely about the label. Four brands, gpt-4o, and the pick itself is judged by gpt-4o.