What happens in the seconds after a prompt
The model decides it needs fresh sources. It picks candidates from a search index, then sends its live agent to fetch each one. The agent identifies itself with a user agent string such as ChatGPT-User/1.0, makes a plain HTTP request and reads the response body.
There is no browser. Client-side frameworks that ship an empty shell and fill it with JavaScript return almost no text to the agent.
A real example: genspark.ai
On 2 Oct 2026 our probe fetched genspark.ai. A browser received HTTP 200 and 206,972 bytes. ChatGPT-User, Claude-User and Perplexity-User all received HTTP 403. The page buyers see in a browser is not the page an AI agent can read.
The two-fetch diff
To see the gap, fetch the same URL in the same second twice: once as a rendering browser and once as each agent. Compare status codes, byte counts and visible text. The difference is what the model never sees.
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)" https://example.com/pricing | wc -cHow to fix a failed fetch
- HTTP 403 or 429: allow agent user agents at the CDN and bot manager.
- HTTP 200 with little text: server-render the content or prerender the route.
- Text present but deep in the document: move content above large inline scripts.
- Redirect stub: serve the final page directly to agents.