BULL AILive fetch
Live fetch: how AI reads your site at the moment someone asks.
Live fetch is when an AI assistant fetches a web page at the moment a user asks a question, instead of answering from training data. ChatGPT does it with an agent called ChatGPT-User, Claude with Claude-User, and Perplexity with Perplexity-User. The agent reads the raw HTML your server returns and does not run your JavaScript. Whatever it can't read in that response, it can't cite.
Why buying questions trigger a live fetch.
Training data is months old. Prices, plans, comparisons and vendor shortlists change faster than that, so when someone asks a buying question, the assistant goes and gets the page. That fetch is the moment your visibility is decided.
Three kinds of AI crawlers, three different jobs.
| Class | Examples | What it does |
|---|---|---|
| Agent (live fetch) | ChatGPT-User, Claude-User, Perplexity-User | Fetches pages when a person asks a question. This is what we measure. |
| Search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Decides whether your pages are candidates to be retrieved. |
| Training | GPTBot, ClaudeBot | Collects pages for future models. |
Gemini is different. It relies on Googlebot, which does render JavaScript, so a site can look fine in Gemini and be unreadable everywhere else.
What breaks a live fetch.
Edge blocks.
A CDN or firewall answers the agent with a 403 or a challenge page. Since July 1, 2025, Cloudflare has blocked AI crawlers by default on new domains.
JavaScript shells.
The raw HTML is an empty container, and your content only appears after scripts run. The agent never runs them.
Content that arrives too late.
Navigation, scripts and markup push the real text deep into the response. We measure how much of your text arrives early.
robots.txt isn't the control it used to be.
On December 9, 2025, OpenAI removed the robots.txt compliance language for ChatGPT-User.
Why live fetch makes AEO fast.
There's no crawl queue and no index refresh. The agent reads whatever your server returns at that instant. Fix the response, and the very next fetch reads the fix. Answer share is a slower clock: it moves over weeks to a quarter as engines re-answer.
The fastest AEOHow to check your own site.
Fetch the same URL twice, in the same second: once as ChatGPT-User with no JavaScript, once in a real browser. Compare the text. That's the two-fetch diff, and it's the first thing your console shows.
How we measure.
Our probe fetches your pages with the agent user agents from our own servers. Some firewalls verify bots by IP address, so when an edge challenges our probe, we report exactly that: your edge challenged our probe. Every result is saved with its date, the agent, and the text received.
In 802 saved probe runs across 113 sites (Aug 17, 2026), ChatGPT-User got a 403 on 96 runs. A browser hitting the same pages got a 403 on 7. Probes ran at the same moment, so some 403s may reflect rate limiting.
Questions about live fetch.
Does ChatGPT run JavaScript when it reads my site?
No. ChatGPT-User reads the raw HTML your server returns. Content that only appears after JavaScript runs isn't read.
What's the difference between GPTBot and ChatGPT-User?
GPTBot collects pages for training future models. ChatGPT-User fetches pages live when someone asks ChatGPT a question. Blocking GPTBot is a policy choice. Blocking ChatGPT-User removes you from live answers.
Does robots.txt control ChatGPT's live fetches?
Not reliably. On December 9, 2025, OpenAI removed the robots.txt compliance language for ChatGPT-User, describing its fetches as user-initiated. Edge rules by crawler class are the real control.
Does Gemini use live fetch?
Gemini relies on Googlebot, which renders JavaScript, so Gemini can see pages the other assistants can't.
How fast does a live-fetch fix show up?
At the next fetch after it ships to production, because the agent reads your page at question time. Answer share moves more slowly, over weeks to a quarter.