Recommendation Intelligence Research™ · Study #28

The model had real web search. It never once reached for it.

Fact Injection showed that when a verified fact is placed directly in a model's context, mention rate jumps by an average of 77.9 percentage points, proof that possession becomes deployment once retrieval succeeds. This study asks the harder question: with real web search actually available, does the model go find that fact on its own? Same 4 underperforming brands, same facts, same 20 buyer questions, but gpt-4o had OpenAI's live web_search_preview tool switched on instead of a hand-placed fact. Across 960 real API calls, it invoked the search tool zero times.

Atom Foundry · September 2026 · Bridges the simulated ceiling in Fact Injection and reality, reuses the same 4 brands, facts, and prompts, part of the Recommendation Intelligence Research™ series

Cite this study: 10.5281/zenodo.22755046

0 of 960Real calls that invoked search

2 of 3Where the same tool did fire

4Brands tested, same cohort

77.9ppInjected ceiling, for reference

Where this comes from

A bridge, not a replication

Fact Injection simulated a successful retrieval: a real, verified fact about a brand was manually placed into the system message, as if a retrieval step had just worked. That proved the "linkage" half of the story, once a fact is in context, the model uses it and mentions the brand far more often. What it could not answer is whether real retrieval actually gets that fact into context in the first place. This study closes that gap directly: same 4 brands, same facts, same 20 already-published buyer-question prompts, but this time nothing is hand-placed. gpt-4o is called through OpenAI's Responses API with the web_search_preview tool switched on, and whether it finds and uses the fact is entirely up to the model.

That gives three directly comparable points on the same axis for each brand: the published baseline (no search, no injection, memory only), this study's live-search condition (real search available, nothing hand-placed), and Fact Injection's injected ceiling (fact hand-placed, simulating perfect retrieval). Baseline and ceiling data are both reused unchanged from Study #23 and Study #25. Only the live-search condition is new.

Model: gpt-4o, via OpenAI Responses API

Tool: web_search_preview, real web access

Brands: 4, same floor-case cohort as Fact Injection

Live-search calls: 800 (4 × 20 prompts × 10 repeats)

Confirmatory re-run: 160 more calls, 2 repeats/prompt

Total real API calls: 960

Why the exact same cohort. Reusing Fact Injection's brands, facts, and prompts means no new design choices are hiding in the comparison, the only variable that changed between the injected condition and this one is how the fact gets into context: hand-placed versus the model retrieving it itself. Brand identities are anonymized throughout this page (Brand A-D), matching Fact Injection's own labeling for the same 4 businesses.

The finding, part one

Available is not the same as used

The first result was not a small effect, it was a flat zero. Across all 800 live-search calls, then again across a separate 160-call confirmatory re-run, the model never once invoked the search tool for these prompts. That is a strange enough result that it deserved active suspicion rather than a quick writeup, a null this clean can just as easily mean "the model chose not to search" as "the citation-extraction code is broken." A 3-prompt diagnostic, run before trusting the main result, settled which one it was: the same tool, same model, same API call structure, but with two of the three prompts rephrased to be explicitly time-sensitive ("...right now in 2026") instead of an open buyer question. Two of those three fired the search tool immediately, each returning ten real citations. The tool works. It simply was never reached for by this exact style of question.

Search-tool invocation rate, by prompt style

Same model, same tool, same API call · only the prompt phrasing differs

Open buyer questions (this study, 960 calls)

0 of 960

Zero invocations

Explicit real-time framing (diagnostic, 3 calls)

2 of 3

10 real citations, each time

This is a finding about tool-choice, not a finding about tool function. "For Shopify, a good post-purchase survey tool is" never triggered search across every phrasing tested in this cohort. "What's a good coffee roaster to buy specialty beans from online right now in 2026?" did, and returned ten grounded citations. The model appears confident enough answering typical open buyer questions from memory that it does not feel a need to check, a decision this series has repeatedly shown to be overconfident on the accuracy of individual facts even when it is broadly right about categories.

The finding, part two

Mention rate barely moved. In three of four brands, it fell.

With search never actually firing, the live-search condition is functionally identical to the baseline condition for these calls, both are the model answering from memory, and the numbers bear that out. Brand D, the one true 0% baseline, stayed at 0%, flat. The other three did not hold steady, they went slightly backward: Brand A moved from 1.25% to 0.5%, Brand B from 5.75% to 2.5%, Brand C from 16.5% to 13.5%. None of that is a meaningful search effect, it is the ordinary sampling noise of asking the same open questions again on a different day. What matters is the direction it did not move: nowhere close to the 96-97.5% ceiling Fact Injection found waiting once a fact actually lands in context.

Live-search mention rate, by brand

Real search enabled, 10 repeats × 20 prompts per brand · tag shows published baseline → Fact Injection's injected ceiling, for reference

Brand C · n=200 calls

13.5%

16.5% → 96%, realized -3.8%

Brand B · n=200 calls

2.5%

5.75% → 97%, realized -3.6%

Brand A · n=200 calls

0.5%

1.25% → 97.5%, realized -0.8%

Brand D · n=200 calls

0% → 44.5%, realized 0%, flat

"Realized ceiling" here means (live rate − baseline) ÷ (injected rate − baseline), and three of four brands land below zero. That is the honest number for what real retrieval delivered against the same theoretical ceiling Fact Injection measured: not a partial recovery, a rounding error around the baseline it started from. The 33 cells where the model did mention a brand were checked for whether the specific fact appeared at all, since search was never invoked and returned zero citations in every one of the 960 calls, any fact that showed up came from memory, the same closed-book knowledge Possession vs Deployment already measured, not from anything retrieved live. 12 of those 33 cells (36.4%) used the fact, a number worth reporting for completeness but too thin a sample, and too disconnected from actual search, to read as a meaningful signal on its own.

Why it matters

Switched on is not the same as used

Web Search vs AI Recommendations, elsewhere in this series, found that 77% of recommendations changed once search was toggled on for a model. That result still stands, it describes what happens once the model does search. This study's contribution sits one step earlier and is more specific: for the exact prompt style used throughout this entire research series, open, unprompted buyer questions with no explicit request to look anything up, gpt-4o never chose to search at all, not once in 960 real calls. The gap Fact Injection measured between possessing a fact and deploying it is real and fixable by getting the fact into context. But for a brand hoping live search closes that gap on its own, the finding here is blunter: the model has to decide to look first, and for this question style, it consistently decided not to.

What this doesn't prove

Scoped to one exact prompt style

This is not a claim that gpt-4o via the Responses API never searches, the diagnostic proves the opposite, differently phrased, explicitly time-sensitive prompts triggered real search with real citations two out of three times. The finding is scoped precisely to the open buyer-question phrasing used across this entire series ("What's a good X for Y?"), and should not be generalized past that exact style without testing it. Because search was never invoked, the Discovery-vs-Usage split planned in this study's design (whether a fact that does get discovered via citations also gets used) could not be measured, zero citations were returned across all 960 calls, so this study is scoped to whether search fires and what happens to mention rate, exactly as the pre-registered design document flagged as the fallback if citations never appeared.

Same 4 deliberately underperforming brands as Fact Injection, a floor-case cohort, not a representative sample. The Responses API plus web_search_preview is a different technical mechanism than the Chat Completions API used throughout the rest of this series, model behavior and tool-choice defaults here could change independently of anything measured elsewhere in this research. Baseline and injected-ceiling data are 1-3+ weeks older than this live-search collection, the same caveat that applies throughout this series. Only 33 cells were even eligible for fact-usage judging, far too few to draw a reliable fact-usage rate from, that number is reported for completeness, not as a finding.

Supporting evidence

Ruling out a bug, not just reporting a null

0 of 960 Real calls, zero invoked search

800 calls in the main run, then a separate 160-call confirmatory re-run with fresh repeats, both logged the same way: zero tool invocations, zero citations, in every single call. The re-run exists specifically to rule out a fluke in the first batch before trusting the result.

2 of 3 Diagnostic prompts where it did fire

A separate 3-prompt diagnostic, run before trusting the main result, isolated prompt phrasing as the variable. Two differently worded, explicitly time-sensitive prompts triggered real search with ten citations each. This is what confirms the zero above is a choice, not a broken tool.