Web Search Rewrites 77% of AI Product Recommendations

A push from a data engineer

A few weeks ago we published research showing that store quality explains almost nothing about which brands AI recommends. Public prominence explained far more. Most of the outcome stayed unexplained.

A data engineer named Rami read it and pushed back with a sharp idea. Maybe a chunk of that unexplained variance wasn’t unexplainable at all. Maybe it was the retrieval layer: whatever the model’s internal search decides to surface quietly decides the recommendation. He proposed a clean way to isolate it: run the same prompts with browsing on and off, and measure how much the answers overlap.

One model, one toggle

50 buying prompts across 10 ecommerce categories. 10 runs per prompt per condition, to control for run-to-run randomness. One model, gpt-4o. The only variable we changed was whether web search was enabled. For each condition we collected the set of recommended brands, then measured how many brands appeared in both. We also ran the whole thing against gpt-4o-mini, to separate the effect of search from the effect of using a bigger model.

77% of recommendations changed

On the same model, the overlap between search-on and search-off recommendations was 23%. In other words, 77% of the recommended brands changed when we turned web search on. Same model, same questions, same number of runs. One toggle.

With search off, the model recommends from memory. It names the brands it saw most often during training, which skews heavily toward the famous ones. With search on, it throws most of those out and recommends whatever its retrieval surfaces in the moment. These are not small adjustments to a stable list. They are two largely different lists.

It was the search, not the model

The obvious objection: maybe gpt-4o just recommends different brands than gpt-4o-mini, and we were measuring model differences, not search. So we isolated it. Same model, search on versus off. Then, separately, the effect of model size with search held off.

Changing the model size (mini to 4o), no browsing, moved about 6% of recommendations. Turning on web search, same model, changed 77%. Which model you ask barely matters. Whether it can browse changes almost everything.

It depends on the category

The effect was not uniform. Browsing rewrote recommendations far more in some categories than others. Pets: 88% of recommendations changed. Fitness: 61%. The pattern: the more fragmented and long-tail the market, the more browsing overrides memory. In categories dominated by a few household names, the model’s memory and its search mostly agree. In categories full of small brands, they don’t, and search wins.

Pets, interestingly, has been the most extreme category in every study we’ve run. It was the sharpest in our earlier quality-versus-recommendation work too. Something about that market makes AI recommendations unusually unstable. We don’t fully understand it yet.

What we still can’t measure

Rami’s original point had a second half we haven’t cracked. He suggested a lot of the unexplained variance hides in training-data provenance: how often the model saw a brand, and in what sentiment context. We can see which brands get named. We can’t yet see whether a model absorbed a brand warmly or coldly during training. That remains the biggest black box in the whole thing.

Being recommended by AI is at least two games

Getting recommended by AI is treated as one goal. It isn’t. Being in the model’s memory is a slow, prominence-driven game that rewards brands the model saw a lot during training. Ranking in what the model retrieves is a live game, closer to search, decided at query time by the retrieval layer.

Most AI-visibility tools measure a third thing entirely: whether the AI can see you at all. But visibility is not recommendation. A recommendation made from memory and one made from retrieval can be almost completely different answers to the same question. If you only optimize for one of these, you’re leaving the other on the table, and depending on the user’s settings, it might be the one that actually decides the sale.

How we measured it

All numbers come from a fixed set of 50 prompts, 10 runs each, per condition, measured directly. No estimates. Overlap is measured on the set of distinct recommended brands per intent. The clean search-effect figure uses gpt-4o in both conditions. One caveat worth stating: browsing implementations change over time, so treat the exact percentages as a snapshot of this model at this moment, not a physical constant. The direction and the size of the effect are the point.