Recommendation Intelligence Research™ · Study #41

Rating recommendations don't move when you reword the question. Price recommendations do.

Recommendation Confidence's Method 6 asked the same 16 contested intents five differently worded ways and found the pick never moved, 0 of 16 flipped, only the hedge language shifted. That page's own limitations section flagged the gap directly: the design was only ever run on the rating signal, price and specs were untested. This study closes it, same 16 intents, same 5 phrasings, same 10-runs-per-phrasing design, run on the price and specs fact pairs instead. The rating signal's stability does not generalize. Price picks flipped in 7 of 16 intents, a 43.8-point gap versus rating that clears significance after correction (p=0.0225). Specs picks flipped in 2 of 16, numerically double rating's rate but not statistically significant at this sample size.

Atom Foundry · September 2026 · Direct follow-up to Recommendation Confidence, closing the exact gap flagged in that page's own “What this doesn't prove” section. Part of the Recommendation Intelligence Research™ series.

Cite this study: 10.5281/zenodo.23350256

+43.8ppPrice vs. rating flip-rate gap (p=0.0225, significant)

0Rating intents flipped, of 16 (published)

2Specs intents flipped, of 16 (not significant)

7Price intents flipped, of 16 (significant)

Where this comes from

A disclosed gap, closed with the exact same design, not a new one

Recommendation Confidence's own limitations section said it plainly: “Methods 5 through 7 do not repeat this design for the price or specs signal types, so the cross-model and phrasing-robustness findings are demonstrated for the rating signal specifically, not confirmed identically for every signal type in this series.” That's a testable claim, not just a caveat, and Method 6 (phrasing perturbation) was the cheapest of the three to extend. So this reuses the identical 16 contested intents from Candidate Evaluation's phase 3 run, the identical 5 phrasing templates, and the identical 10-runs-per-phrasing design, the only change is loading the price-signal and specs-signal fixed fact pairs instead of the rating-signal ones. Nothing new was generated, no scenario was reshuffled.

Model: gpt-4o, temperature 0.7

Design: 16 contested intents × 5 phrasings × 10 runs, per signal

Total calls: 1,600 (800 price, 800 specs)

Pick extraction: majority vote of 10 runs per phrasing, identical rule to published Method 6

Statistics: cluster-permutation test, unit=intent (n=16 per signal), 10,000 permutations

Correction: Bonferroni ×3 (price vs. rating, specs vs. rating, price vs. specs)

Same source file, same scenarios, only the signal filter changes. Both new CSVs are generated from the exact same phase3_raw.csv Recommendation Confidence already used, filtered to signal=price or signal=specs, a_had_better_fact=True, run=1 instead of signal=rating. The rating-signal comparison number below (0 of 16) is the already-published result, reused unchanged, not rerun for this page.

One real example

Same facts, five phrasings. Four of five land on a different brand than the first.

Best cat litter for odor control, Arm & Hammer versus Dr. Elsey's, Dr. Elsey's holding the better fixed price fact. Here is exactly what each phrasing's 10 runs returned.

Baseline

Dr. Elsey's, 7 of 10 · Arm & Hammer, 3

Casual

Arm & Hammer, 10 of 10 ← flips

Formal

Arm & Hammer, 8 of 10 · Dr. Elsey's, 2

Reversed order

Arm & Hammer, 7 of 10 · Dr. Elsey's, 3

Imperative

Arm & Hammer, 9 of 10 · Dr. Elsey's, 1

This is not a coin-flip artifact. Every one of these five majorities is decisive, 7-3 at the loosest, 10-0 at the tightest, none of the 10-run splits land anywhere near an even 5-5. The baseline phrasing alone favors Dr. Elsey's; every other phrasing swings hard to Arm & Hammer. That is exactly the kind of flip the published rating-signal Method 6 reported zero of, across all 16 intents and all 5 phrasings. Not every price flip in this batch is this clean, two of the seven (best air purifier for allergies, in two of its five phrasings) hinge on an exact 5-5 tie in the 10-run vote, which the same majority-pick rule used throughout this series resolves arbitrarily. That caveat is carried into the limitations section below.

The finding

Rating holds. Specs mostly holds. Price doesn't.

Same question as the published Method 6, asked of two more signal types: across the 16 contested intents and 5 phrasings, how often does the majority-vote pick change depending on which phrasing was used.

Share of intents where the pick flipped across 5 phrasings, by signal type

16 contested intents per signal · 5 phrasings × 10 runs = 800 calls per signal (rating figure republished from Method 6)

Rating (published)

0.0%

0 of 16

Specs

12.5%

2 of 16, not significant

Price

43.8%

7 of 16, significant

The 7 price flips: pets-02, coffee-02, fashion-05, food-02, home-01, home-05, wellness-05. The 2 specs flips: pets-04, home-05. Rating's 0 of 16, republished from the already-live page, is the baseline every other signal in this series has been measured against for phrasing robustness. It turns out that baseline was specific to rating, not a property of “fixed facts” in general.

Making sure it holds

The cluster-permutation test that decides it

Per the house standard for this series, the primary test treats each intent as the independent unit (16 per signal, not 800 individual calls), permutation-tests the flip-rate difference between signal pairs, and Bonferroni-corrects across the 3 comparisons this page makes.

Comparison	Flip-rate diff	Raw p	Bonferroni-corrected p (×3)	Result
Price vs. rating	+43.8pplargest gap	p=0.0075	p=0.0225	Significant
Specs vs. rating	+12.5pp	p=0.4936	p=1.0	Not significant
Price vs. specs	+31.2pp	p=0.1137	p=0.341	Not significant

Only one of three comparisons clears the corrected bar, and it's the one the hero number rests on. Price's flip rate is significantly higher than rating's even after correcting for testing three comparisons at once (10,000-permutation test, unit=intent, p=0.0225). Specs numerically doubles rating's flip rate, 12.5% versus 0%, but with only 16 intents per signal that difference does not clear significance, p=1.0 after correction. Per this series' no-file-drawer standard, the honest read is: price breaks phrasing stability, specs might but this sample can't confirm it, and price versus specs itself is not a confirmed difference either.

A secondary, counterintuitive finding

The signal that flips the most hedges the least

The same Hyland-style hedge/booster lexicon from the rest of this series, applied to each call's one-sentence reason. If hedge language tracked real instability, price, the least stable signal, should hedge the most. It's the opposite.

Most-hedged phrasing per signal

Peak hedge rate reached by any of the 5 phrasings Rating (published): baseline, 15.6% · 0 of 16 picks flipped
Specs: baseline, 6.9% · 2 of 16 flipped
Price: formal, 3.8% · 7 of 16 flipped

Least-hedged phrasing per signal

Lowest observed hedge rate across the 5 phrasings Rating (published): reversed / casual, 6.9%
Specs: formal, 3.1%
Price: baseline, 1.2%

Rating hedges the most and flips the least. Price hedges the least and flips the most. Every hedge-rate number for price sits below every hedge-rate number for specs, which sits below every hedge-rate number for rating, even though the stability ranking runs the exact opposite direction. Whatever produces hedge language in this model's output, it is not tracking how likely the pick is to move if the question gets reworded. That's consistent with what Recommendation Confidence already found within the rating signal alone, hedge rate and real pick stability are separate variables, and this extension shows the same disconnect holds across signal types too.

Why it matters

The stable-decision finding was never a universal one

Recommendation Confidence's headline, that the model's pick barely moves no matter how the question is phrased, held completely for the rating signal: 0 of 16 flips, 5 phrasings, 10 runs each. It was always fair to ask whether that held for other kinds of injected facts, and it doesn't uniformly. A brand whose advantage is a lower price should not assume a reworded question leaves its recommendation odds where they were, 7 of 16 price-based comparisons in this sample landed on a different brand depending on phrasing alone. A brand whose advantage is a spec claim sits closer to rating's stability, but not confirmed identical to it either, 2 of 16 flipped, a real number even if this sample can't rule out chance. The practical read: don't generalize a phrasing-robustness result from one signal type to another without testing it, this series' own headline finding is the proof that assumption can be wrong.

What this doesn't prove

A closed gap, and two new open ones

The specs result (2 of 16, p=1.0 corrected) is not confirmation that specs behaves like rating, it's an absence of a detected difference at this sample size, a different claim. Twice rating's raw flip rate is not a small number to wave away; a larger sample could plausibly find specs closer to price than to rating. The majority-pick rule inherited unchanged from the published Method 6 breaks 10-run ties arbitrarily by insertion order, and 2 of the 7 price flips and 1 of the 2 specs flips pass through at least one dead-even 5-5 phrasing, meaning some of what's counted as “flipped” may be closer to genuine model indecision than a clean phrasing-driven swing. Cross-model (method 7) and the logprobs/self-report confidence methods (3 and 4) remain rating-only, this page closes the phrasing-perturbation leg of the disclosed gap specifically, not the whole gap. Single model (gpt-4o), single temperature (0.7), same 16 intents and 5 phrasing templates as the rest of this sub-series, not independently re-designed for price and specs.

Everything here is correlational and observational within a controlled prompt design, not a causal test on a real store, the same standing limitation as every study in this series. It measures whether a model's stated pick moves when the question's wording changes, holding the underlying facts fixed; it does not measure what happens if a real brand's own price or spec claims change in the wild.