Candidate Evaluation found that a better rating decides a contested pick almost every time. This study asks the next question: when the model explains that pick, does its language, hedged ("seems," "may be") versus assertive ("clearly," "significantly"), actually track how confident it really is? Seven independent methods test this on the same 16 fixed scenarios: a hedge/booster lexicon, a blind second-model judge, real token logprobs, self-reported confidence, repetition stability, five ways of phrasing the same question, and replication across GPT-4o, Claude Sonnet and Gemini. The answer is consistent across all seven: verbal hedging correlates r=0.03 with real internal confidence, and for the pick type that draws the most hedged language, real confidence is the highest of any signal tested.
How AI Decides names Confidence as Stage 9 of the decision path, open research at the time: "how certain is AI about the recommendation, and how much would the answer change with slightly different phrasing?" Candidate Evaluation had already shown that a better star rating decides a head-to-head comparison almost every time. That leaves an obvious question unanswered: when the model writes out its reasoning for a pick, does the language it reaches for, hedged phrasing like "seems" or "may be" versus assertive phrasing like "clearly" or "significantly", actually reflect how sure it is internally? Or is hedging just a writing habit, disconnected from what is really going on underneath?
To answer that cleanly, every method here reuses the exact same 16 fixed contested-pair scenarios from Candidate Evaluation's phase 3 run: the same brand_a/brand_b pairs, the same injected fact_a/fact_b, the same rating-signal setup where one brand has a strictly better rating. Nothing new is generated. Seven different measurement techniques are simply pointed at the identical underlying decisions, so every method's numbers are directly comparable row for row.
Each method attacks a different weak point in the one before it. Together they triangulate on a single answer instead of resting on one technique's blind spots.
Candidate Evaluation tested three signal types: price, rating, and specs. The lexicon classifier (method 1) found the rating signal draws hedged language far more often than the other two, 11.2% of responses versus 2.5% for price and 3.1% for specs, a statistically significant gap (p=0.03 vs. price). The obvious reading is that the model is genuinely less sure when a rating is doing the deciding. Token logprobs (method 3), the actual probability gap between the chosen brand and the alternative, say the opposite: the rating signal has the highest real confidence of the three, not the lowest.
A hedge/booster word list is a blunt instrument, so before leaning on it, a second model (Claude) blind-rated 50 anonymized reasoning texts with no access to the lexicon's labels or word lists, just the raw text. The two only agreed on the exact three-way label (hedged / neutral / assertive) 66% of the time, and on the coarser hedged-vs-not distinction 76% of the time. Most of the disagreement traces to one specific failure mode: sentences that mix a hedge word and a booster word in the same sentence, which the lexicon scores as a false "neutral" tie, but a human or model reader would usually call assertive overall.
Separately, method 5 asks whether hedge language tracks something else entirely: real behavioral instability, whether the model actually picks a different brand across 10 repeated runs of the identical prompt. Across four separate datasets (50-brand and 16-intent recall runs, with and without injected facts), the correlation between hedge rate and how often the top pick actually changes ranges from −0.09 to +0.11, essentially zero in every case.
The same 16 fixed facts were asked five different ways: a plain baseline, a casual tone, a formal tone, the brands listed in reversed order, and an imperative instruction. Across all 800 calls, 0 of 16 intents changed their winning brand no matter which phrasing was used, the underlying decision is completely robust to how the question is asked. The hedge rate, however, moved by more than double depending on phrasing alone, nothing about the facts changed.
The same 16 fixed scenarios were sent, unchanged, to GPT-4o, Claude Sonnet 4.5 and Gemini. Across 480 calls, every one of the 16 intents landed on the same majority pick across all three models, and within each model the pick barely moved across its own 10 runs (0% flip rate for GPT-4o and Claude, 1.2% for Gemini). That is strong evidence the injected facts themselves, not any single lab's quirks, are what drives the decision.
Self-reported confidence, though, has a distinct house style per provider: GPT-4o averages 87.9, Claude the most conservative at 78.8, Gemini the most confident at 91.4 with the widest spread (30 to 100). But the language pattern in each model's stated reason is nearly identical: assertive/boosted language dominates everywhere, 87.5% of GPT-4o's reasons, 93.8% of Claude's, 81.9% of Gemini's, with true hedge language rare across the board (0.6% to 1.9%). The confident-sounding tone isn't a GPT house style, it's close to universal.
Seven independent tests, four different confidence proxies, three different providers, and five different phrasings all land on the same place: how confident or hedged a model's language sounds is not a reliable signal of how confident it actually is. For a brand watching how AI describes it, "the AI clearly recommends us" and "the AI seems to lean toward us" are, on this evidence, not meaningfully different signals of real model conviction, both could be sitting on an internal confidence anywhere in the same range.
What did stay stable throughout: the actual pick. 0 of 16 decisions changed across five phrasings, and 16 of 16 agreed across three separate model providers. That is the practical takeaway for the rest of this series: the lever that moves a recommendation is the fact injected into the comparison (a rating, a price, a spec), not the tone the model happens to write in afterward, and not which provider is asked. Chasing "sound more confident" copy is chasing the wrong variable.
Stating the limits up front. All seven methods hold the same variable fixed: a rating-signal, contested-pair scenario where one brand has a strictly better rating (reused from Candidate Evaluation's phase 3). Methods 5 through 7 do not repeat this design for the price or specs signal types, so the cross-model and phrasing-robustness findings are demonstrated for the rating signal specifically, not confirmed identically for every signal type in this series.
The lexicon classifier (method 1) that most of the headline percentages rest on only reaches 66% three-way agreement with an independent model judge, treat any single hedge-rate percentage as directional. Token logprobs, the one genuinely unfakeable confidence measure, are only available through OpenAI's API, Anthropic does not expose them and Gemini's availability is inconsistent across API versions, so methods 3's "real confidence" comparison is GPT-4o only; Claude and Gemini are represented only through self-report, which method 4 already shows is an imperfect proxy (r=0.345 vs. real logprobs). And Gemini's leg of method 7 ran on a Flash-tier model rather than the originally planned Pro-tier model, for the free-tier quota reason noted above.
Everything here is correlational and observational within a controlled prompt design, not a causal test on a real store. It measures how a model talks and how sure it internally is about a fixed, artificial comparison; it does not measure whether changing a real brand's language in the wild would change how AI recommends it. That causal question is the direct motivation for the Founder Lab experiments noted on the research roadmap.
Even the model's own stated 0-100 confidence number only weakly tracks its real internal signal (token logprobs). It doesn't fully know its own mind, so the words it chooses on top of that number track real confidence even less.
A second model (Claude), blind to the lexicon's labels, agreed on the exact hedge/neutral/assertive rating only 66% of the time on 50 sampled texts. Built into every hedge-rate number in this study as the honest error bar.
0 of 16 picks flipped across 5 phrasings of the same question, and all 16 stayed identical across GPT-4o, Claude Sonnet and Gemini. The decision is the stable part of this system; the wording around it isn't.
If a model's confident-sounding language doesn't track its real confidence, a free scan is a faster way to find what actually moves a recommendation than parsing how AI describes you.
This answers the Confidence question flagged on the decision map. Read the map for where it fits, or Candidate Evaluation for the original rating-signal decisions this study reuses.
Illustrative example · single-site signal for atomfoundry.dev.
View full signals →