GPTBot is OpenAI's crawler for collecting training data, and ClaudeBot is Anthropic's. Both obey robots.txt, and blocking them only opts you out of training. To appear in AI answers, the crawlers that matter are the search and user crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. So most sites that want AI visibility should allow those, and decide separately about training.
There's one thing most guides leave out: many requests that claim to be these bots aren't. In my own server logs, only 20% of requests calling themselves GPTBot came from OpenAI's published addresses.
Which AI Crawlers Exist
AI companies now use separate crawlers for separate jobs. Blocking each one costs you something different.
| Crawler | Company | Job | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Training data | Your content isn't used to train OpenAI's models |
| OAI-SearchBot | OpenAI | ChatGPT search | You won't be shown in ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetches a page a user asked about | OpenAI says robots.txt rules may not apply |
| ClaudeBot | Anthropic | Training data | Your future content is left out of Claude's training data |
| Claude-SearchBot | Anthropic | Claude's search | Less visibility and accuracy in Claude's search results |
| Claude-User | Anthropic | Fetches a page a user asked about | Claude can't retrieve your pages for users |
| PerplexityBot | Perplexity | Perplexity search | You won't appear in Perplexity's results |
| Perplexity-User | Perplexity | Fetches a page a user asked about | Perplexity says this fetcher generally ignores robots.txt |
| Google-Extended | A control token, not a crawler | Opts out of Gemini training and grounding. Google says it doesn't affect Google Search | |
| Applebot-Extended | Apple | A control token, not a crawler | Opts out of Apple's AI model training |
| CCBot | Common Crawl | Open web archive many AI models are trained on | Your pages are left out of future Common Crawl archives |
Sources: OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl. The robots.txt guide explains how to write the rules.
What Visited My Sites
I counted requests in the raw access logs of three of my sites, ivig.dev, mouse-jiggler.org and a product blog, from 1 to 12 November 2026. Then I checked each request's IP address against the address lists each company publishes.
| Claimed crawler | Requests | From the published addresses |
|---|---|---|
| GPTBot | 522 | 109 (20%) |
| ChatGPT-User | 612 | 169 (27%) |
| OAI-SearchBot | 816 | 365 (44%) |
| PerplexityBot | 776 | 376 (48%) |
| Googlebot | 1,947 | 667 (34%) |
| Bingbot | 1,499 | 1,014 (67%) |
I looked up the addresses that failed the check. The busiest ones were rented Google Cloud servers, with host names ending in googleusercontent.com, not Google's or OpenAI's crawlers. The same addresses claimed to be Googlebot and GPTBot. They were scrapers pretending to be well-known bots.
Two limits on these numbers. The address lists are today's, so a few genuine requests from addresses a company has since retired may count as unverified. And ClaudeBot and Perplexity-User couldn't be checked this way: Anthropic's help page doesn't list crawler addresses, and Perplexity's list for Perplexity-User held only ten. On ivig.dev in October, requests claiming to be ClaudeBot (1,393) outnumbered those claiming to be Googlebot (953), but I can't say how many of either were genuine.
How to Tell a Real Crawler From a Fake
The user agent is just text, and anyone can send it. Check the IP address instead:
- OpenAI publishes address lists for each crawler:
openai.com/gptbot.json,openai.com/searchbot.jsonandopenai.com/chatgpt-user.json. - Perplexity publishes
perplexity.ai/perplexitybot.jsonandperplexity.ai/perplexity-user.json. - Google publishes its crawler addresses on its verifying Googlebot page. A reverse DNS lookup on a real Googlebot address ends in
googlebot.comorgoogle.com. - Bing publishes
bing.com/toolbox/bingbot.json. - Common Crawl warns that crawlers falsely identify themselves as CCBot, and recommends verifying them.
Why it matters: robots.txt only works on crawlers that choose to obey it, and impostors don't. If fake bots are loading your server, block them by address or with your host's bot protection. Changing robots.txt won't stop them, and blocking by user agent would block the real crawlers too.
What to Allow and What to Block
| Your goal | Allow | Block |
|---|---|---|
| Be cited in AI answers, and don't mind training | Everything (the default) | Nothing |
| Be cited in AI answers, but stay out of training | OAI-SearchBot, Claude-SearchBot, PerplexityBot, and the user fetchers | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot |
| Keep content away from all AI products | Googlebot and Bingbot for normal search | All AI tokens. User fetchers may still visit, because some ignore robots.txt |
For most businesses that want to be found, I'd use the first or second row. Blocking the search crawlers removes you from the answers your customers may be reading. My own sites allow everything.
The second row as robots.txt rules:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlOpenAI says robots.txt changes take about 24 hours to affect search. Blocking a crawler doesn't remove content already collected. If an AI crawler needs your content, it also needs to see it in the HTML: most AI crawlers don't run JavaScript. For what else helps you get cited, see how to rank in ChatGPT.
Check Your Own Logs
Most hosts keep raw access logs. In cPanel, look for Raw Access in the Metrics section. This command counts requests per claimed AI crawler in a log file:
grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Googlebot|bingbot' access.log \
| tr 'A-Z' 'a-z' | sort | uniq -c | sort -rnFor a compressed monthly log, start with zcat access-log.gz | instead. Treat the result as an upper limit until you've checked the addresses against the published lists.
Want to Know What AI Tools See on Your Site?
My AI search audit ($255, 2-day delivery) checks whether AI crawlers can reach and read your pages, tests how AI tools describe your business, and gives you a prioritized list of fixes. For a first look, request a free website review.
Request a free website reviewFrequently Asked Questions
Related Guides
- Write robots.txt rules correctly.
- See what gets a site cited in ChatGPT.
- Check whether AI crawlers can see your content.
- What Google's Open Knowledge Format means for AEO.

