Research / Mechanism Studies

The mechanism behind the headline finding

The flagship report showed that store readiness does not predict recommendation. These 40 studies dig into why, each one testing a single variable such as web search, brand fame, domain recognition, self justification, vocabulary, intent, the model's own memory, what tips the verdict once two candidates are genuinely close, how a brand with no memory gets considered at all, or whether the model's language reflects its real confidence.

21 published studies

Isolating one variable at a time

Each study holds the model and the prompts constant and changes exactly one thing, so the effect can be attributed with confidence rather than guessed at.

Web Search Rewrites 77% of AI Product Recommendations

Turning on browsing changes 77% of recommended brands, same model, same prompts.

77% of brands changed

Mechanism, Corrected

The Fame Study, Corrected

Public fame explains 1.2% of recommendation, re-measured on 872 brands, sitting on the noise floor.

AI Knows Your Website. It Still Won't Recommend You.

The model names the correct domain 75.9% of the time. Knowing where you are is not the same as reading your store.

75.9% domain accuracy

29,633 Reasons. 26,812 Unique. The Model Confabulates.

The model writes a fresh justification for every pick. Asking it why does not work.

90% unique reasons

Search Changes the Vocabulary, Not Just the Brands

With search on, the model names ingredients and specifications. With it off, impressions.

21x more specific

Candidacy vs Selection

Full population, 60,924 stores. Intent gets you into the recommended set. It wins you nothing once you're there.

Nothing About Your Brand Predicts Recommendation. The Model's Own Past Behavior Does.

June's position predicts July's position, the strongest signal in this entire series, and it does not drift for 15 days.

61.4% self-predictive

Mechanism: Follow-up

Two Months Later, the Model Still Agrees With Itself

50 intents, six independent sweeps across two months. 43 kept the exact same #1 brand every time.

Hand It a Rating, and It Follows Every Single Time

16 genuinely contested brand pairs. A better star rating decided the pick in 160 of 160 runs, zero exceptions.

100% follow rating

We Invented a Brand With Zero History. Reviews Got It Picked Anyway

Against an entrenched category leader, a brand with no memory won 0 of 360 runs with no evidence, 53.1% once given a review score.

0% to 53.1% with reviews

We Widened the Fame Signal Four Ways. It Barely Moved

Wikipedia, Wikidata, domain age, and media mentions combined explain 11.2% of recommendation frequency, still far under the 61.4% self-consistency benchmark.

11.2% combined, vs 61.4%

The Model Hedges Most When It's Most Sure

Seven independent methods find hedged language correlates r=0.03 with real internal confidence, and is highest exactly where real confidence is highest.

r=0.03, 16/16 cross-model agreement

The Model Knows 6 Facts About Your Brand. It Uses One

A direct possession probe scored against 4,000 real buyer-question responses: 75.8% of possessed facts never appear in an actual recommendation.

75.8% gap rate, 9 brands

It Recommends You First. By Turn Four, It's Moved On

200 four-turn conversations, explicit favorable intro then two brand-blind follow-ups: 49.5% survive to the forced final pick, ranging 100% to 0% by brand.

49.5% T4 survival, 10 brands

Give the Model the One Fact It's Missing. The Brand Goes From Invisible to Everywhere

One real, verified fact injected into context for 4 underperforming brands, simulating successful retrieval: average mention-rate lift of 77.9 percentage points.

+77.9pp avg lift, 4 brands

Tell the Model You Saw an Ad. It Recommends That Brand 88% of the Time

Controlled manipulation of hidden system-message context across 5 brands: a fabricated ad-exposure claim alone beat a combined brand+real-fact condition for winner-rate lift.

88% winner rate, fabricated ad condition

The Model Is Almost Never Wrong About Your Brand. It Just Doesn't Say Much

Closed-book claims independently fact-checked against real sources instead of self-graded: 97.1% of checkable claims held up, but unprompted topic coverage averaged only 61.1% across 8 real attribute categories.

97.1% accurate, 61.1% depth coverage

The Model Had Real Web Search. It Never Once Reached for It.

Same 4 brands and facts as Fact Injection, but with real web search enabled instead of a hand-placed fact: 0 of 960 real calls invoked the search tool. A diagnostic confirms the tool works, just not for this prompt style.

0 of 960 calls invoked search

We Described One Brand to the Model, Real or Invented. It Recommended That Brand Anyway.

Five escalating rounds, 1,997 real calls: 12 of 13 brands stayed near 100% candidacy no matter how the framing changed, and even a fully invented brand hit the same ceiling.

12 of 13 brands, no framing effect

Ask Like You're Googling It, and the AI Recommends the Brand 15.6 Points Less

A 15-cell question length and sentence-type grid plus 7 writer-style personas, 704 calls, same facts throughout: bare-keyword phrasing loses to a full natural-language question by the same 15.6 points in both tests.

15.6pp gap, found independently twice

Say It Yourself, and the Model Picks You 67.5% of the Time. A Third Party Only Gets 51.1%.

Same fact, same length, three apparent sources: the brand itself, a neutral description, an independent third party. Tested twice, 3,200 calls total, round 2 rebuilt to rule out a wording confound. Self-claim still wins most in every brand.

67.5% vs 51.1% vs 30.4%, both rounds

Specific Claims Add 15.7 Points in Spec Driven Categories. They Cost Nearly 5 in Trust Driven Ones.

Same real facts, rewritten as vague marketing copy or as concrete, PDP style claims. 8 brands, 3 rounds, 3,600 calls: specificity adds 15.7 points in purely functional categories and costs close to 5 in trust and safety sensitive ones.

+15.7pp functional, -4.5pp trust, 8 brands

Give It the Better Rating, and It Wins 91% of the Time

Rating, claim specificity, and message format, all tested at once for the first time, competitor also carrying real information. 4 brands, 2 rounds, 6,400 calls: rating decides the winner 90.8% of the time regardless of the other two.

90.8% follows rating, +5.8pp specificity

Mention It Was Featured Somewhere, and It Wins 85% of the Time. Bring In a Rating, and It Nearly Disappears.

Winner vs Loser's own named follow-up: a disclosed-synthetic third-party mention, tested alone and then crossed against rating in a full 2x2. 4 brands, 2 phases, 2 rounds, 4,800 calls: strong alone, almost swamped once rating is in the room.

85.2% alone, +1.5pp once rating disagrees

Claim the Brand Is Widely Known, and It Wins 80% of the Time. Bring In a Rating, and the Edge Is Nearly Gone.

Authority's direct sibling: a bare, unsourced claim of broad recognition, no named third party, tested the same two-phase way. 4 brands, 2 phases, 2 rounds, 4,800 calls: strong alone, the most completely swamped signal measured against rating so far.

79.8% alone, +0.2pp once rating disagrees

The Cited Source Changes. The Winner Almost Never Does.

Live Retrieval's direct follow-up: real product categories, open-ended prompts, live search on. 8 categories, 2 rounds, 320 calls: winner stability held near 90% even as which exact page got cited moved far more.

90% winner stability, diff not significant (p=0.25)

Two Signals Get You Most of the Way There. A Third Barely Helps, and Format Doesn't Move It at All

Synthesizes winner-vs-loser, authority-signal, brand-familiarity, and pdp-specificity: authority, familiarity, specificity, and format tested against each other with rating removed entirely. 12 conditions, 2 rounds, 9,600 calls: specificity leads, format's own effect rounds to zero, two signals already get you most of the way there.

+3.25pp specificity, strongest marginal signal

We Turned the Same Facts Into Bullets. The Model Cited Fewer of Them.

A third, most direct test of the format signal: same 3 facts per brand, only the separator changes, prose paragraph vs bulleted list. 4 brands, 2 rounds, 1,600 calls: structured format cites fewer facts and reuses less vocabulary than prose, not more.

-5.2pp citation rate, structured vs prose

Authority Adds 9 Points in Trust Categories. Familiarity Adds 14.

The direct counterpart to PDP Specificity: reruns authority and familiarity on 8 brands, 4 functional, 4 trust sensitive, splitting the result by category from the start. Opposite pattern from specificity, social proof helps trust categories more, not less. 6,400 calls, 2 independent rounds.

+9.1pp authority, +13.8pp familiarity, trust vs functional

Reword the Question Three Ways. The Winner Mostly Stays the Same.

Direct follow-up to Source Stability, testing a specific LinkedIn objection: does stability depend on wording, not just literal repeats. Same 8 categories, 3 phrasings each, 2 independent rounds, 240 live search calls. Not significant in either round alone.

+3.1pp combined, p=0.063, not confirmed

Rating Recommendations Don't Move When You Reword the Question. Price Recommendations Do.

Closes the price/specs gap Recommendation Confidence's own limitations flagged. Identical Method 6 design rerun on price and specs signals, 1,600 gpt-4o calls. Rating's published 0-flip result does not generalize.

7/16 price flipped (sig.), 2/16 specs (not sig.)

A Bad Rating Barely Touches Candidacy. It Wrecks the Final Round.

Tests a LinkedIn commenter's claim that AI pre-filters bad actors before they're ever mentioned. 4 brands reused from Winner vs Loser, neutral vs. negative reputation, two probes. The result is the opposite of pre-filtering.

Candidacy +15.5pp, selection -59.2pp

Mark the Listing Sponsored, and It Loses 18.4 Points in a Head-to-Head Pick

Adjudicates two conflicting outside numbers, Zhao Hanbo's near-zero ad effect vs. Allouah et al.'s active penalty. Same 4 brands, one label swapped: standard result vs. Sponsored result, forced two-way comparison.

Organic 74.2%, sponsored 55.8% (-18.4pp)

Match the Sponsored Listing to the Search Term, and the Penalty Overshoots by 21 Points

Direct follow-up to Sponsored Tag Penalty. One sentence echoing the exact search term into a sponsored listing didn't just recover the 18.4pp penalty, it beat the organic baseline outright.

Static 55.8% -> query-matched 95.2% (+39.4pp)

Rank First in the Cited Source, and the Model Always Agrees. Fourth Place Wins 1.2% of the Time.

Inspired by a LinkedIn exchange with Olga Vainshtok. Same 4 brands, one synthetic 17-slot roundup source, only the target's rank inside it moved. Candidacy barely drops with position. Selection collapses to almost nothing past rank 1.

Selection 100% (rank 1) -> 1.2% (rank 4) -> 0% (rank 15)

Under 4.1 Stars, the Model Ignores an Unknown Brand. At 4.9 Stars It Picks It 85% of the Time.

Direct follow-up to Cold Start. 36 intents, 11 fixed doses of review evidence for an unknown brand against the category's default pick. The rating is the lever and has a threshold, the review count decides whether the model believes it. Rating alone wins 26.4%, count alone 2.5%.

Selection 2.2% (4.1 stars) -> 18.1% (4.4) -> 84.7% (4.9)

In a Ranked Source, the Higher-Ranked Brand Wins 100% of the Time, Even Against the Model's Favorite.

Position, not brand, decides inside a ranked source. An unknown brand ranked #1 beats the model's favorite 100%, ranked one place lower 0%. No gradient across 14 places; instruction, placement and real brands change nothing; the same list unordered gives 0.8%.

Ranked #1 100% · ranked #2 0% · unordered list 0.8%

Press Coverage Moves the Model Only When It Answers the Exact Question Asked. Generic: 1.9%. Matching: 80.6%.

A press mention does nothing on its own (1.9%). Naming the brand for the exact question asked wins 80.6%; the same endorsement for an adjacent use case or another category wins 0%. Press carrying verifiable facts performs like showing them directly (52.8% vs 51.9%).

Generic 1.9% · exact query 80.6% · adjacent 0%

Describe a Company by Its Side Feature and It Falls Off the AI Shortlist: 82.8% to 24.0%. The Model Still Names Its Category Correctly.

Describe the company by a side feature and it falls off the shortlist: 82.8% when the core leads, 24.0% when the side feature dominates. The model still names the core category correctly (100%), and stating the primary category does not repair it (27.5%). Weight and order of the evidence both matter.

Core leads 82.8% · side feature dominates 24.0% · core named 100%

Reviews Four Years Old Count as No Reviews at All: 0%. Five Hundred Fresh Ones Beat Thirty Thousand Old Ones, 72.8% to 1.4%.

Same rating, same count, only the age of the reviews changes. 500 reviews from the last 3 months: 72.8%. Undated: 50.8%. All posted 4 years ago: 0.0%, the same as no evidence. 30,000 old reviews: 1.4%. A fall in the last 3 months' rating overrides the overall rating.

Fresh 72.8% · undated 50.8% · 4 years old 0.0% · 30,000 old 1.4%

See where your store stands

Free AI Commerce Score across all 8 factors.

Find out exactly what AI agents see when they evaluate your store.

Free · No signup · Results in 10 seconds