Step 1: robots.txt rules
Add these groups to /robots.txt at the root of your domain. GPTBot is optional and only affects training data.
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /Step 2: check the CDN and bot manager
Cloudflare, Akamai, Fastly and similar services can block AI user agents before robots.txt is ever read. Look for an "AI bots" or "AI crawl control" toggle and any custom rule that matches GPTBot or ChatGPT.
In our saved probe of genspark.ai on 2 Oct 2026, a browser received 206,972 bytes while the live AI agents received HTTP 403. That is the pattern to rule out.
Step 3: test with a real request
Send a request with the ChatGPT-User string and compare it with a browser request. Both should return 200 and similar text.
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
https://example.com/Step 4: make sure the fetch contains your content
ChatGPT-User does not run JavaScript. A 200 response that only contains an empty app shell still gives ChatGPT nothing to quote. Server-render or prerender the text that matters.