The five-stage citation chain
Every cited answer passes through five stages. If any stage fails, the later ones cannot succeed.
| Stage | Question | What moves it |
|---|---|---|
| 1. Candidate | Is the page in the retrieval set? | Search-class crawler access, indexation, Bing coverage |
| 2. Fetch | Does the live agent get HTTP 200? | robots.txt, CDN and WAF rules for agent user agents |
| 3. Read | Is the text in the raw HTML? | Server rendering, smaller inline scripts, content early in the document |
| 4. Survive | Do key sentences survive truncation? | Main text ratio, first content offset, concise answer-first passages |
| 5. Cite | Does the answer name and link you? | Distinct claims, comparison pages, third-party mentions |
Three crawler classes, three jobs
Agent class (ChatGPT-User, Claude-User, Perplexity-User) fetches pages live while a person is asking a question. This is the class we treat as the unit of measurement.
Search class (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot) builds the index that decides candidacy. Training class (GPTBot, ClaudeBot) collects training data and is a comparator, not a lever for live answers. Gemini relies on Googlebot, which does render JavaScript.
Engineering changes that move answers
These are the fixes we see move the fetch most often in diagnoses.
- Unblock agent user agents at the CDN or firewall, not only in robots.txt.
- Server-render pricing, feature lists and comparison tables.
- Move the main content above large inline script and style blocks so it lands inside the agent read window.
- Replace redirect stubs and consent walls that return a near-empty body to agents.
- Keep one canonical URL per topic so citations consolidate.
Content changes that move answers
Once the fetch is clean, content is the second lever. Write answer-first passages: define the thing in the first sentence, give a specific number, name the entity. Models quote sentences that stand on their own.
Publish comparison and alternatives pages for the prompts buyers actually ask, and make sure each claim carries a scope and a date.
Ship, re-probe, repeat
AEO is a loop, not a launch. Bull AI Labs runs it as a sprint: up to 4 tasks per sprint, each tagged engineering, content or crawlability, and each verified by re-fetching the page after it ships. The diagnosis, prompt bench and LLM Lab keep the loop measured.