GPTBot is OpenAI's crawler for collecting training data, and ClaudeBot is Anthropic's. Both obey robots.txt, and blocking them only opts you out of training. To appear in AI answers, the crawlers that matter are the search and user crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. So most sites that want AI visibility should allow those, and decide separately about training.

There's one thing most guides leave out: many requests that claim to be these bots aren't. In my own server logs, only 20% of requests calling themselves GPTBot came from OpenAI's published addresses.

Which AI Crawlers Exist

AI companies now use separate crawlers for separate jobs. Blocking each one costs you something different.

Main AI crawlers and what blocking them does
CrawlerCompanyJobIf you block it
GPTBotOpenAITraining dataYour content isn't used to train OpenAI's models
OAI-SearchBotOpenAIChatGPT searchYou won't be shown in ChatGPT search answers
ChatGPT-UserOpenAIFetches a page a user asked aboutOpenAI says robots.txt rules may not apply
ClaudeBotAnthropicTraining dataYour future content is left out of Claude's training data
Claude-SearchBotAnthropicClaude's searchLess visibility and accuracy in Claude's search results
Claude-UserAnthropicFetches a page a user asked aboutClaude can't retrieve your pages for users
PerplexityBotPerplexityPerplexity searchYou won't appear in Perplexity's results
Perplexity-UserPerplexityFetches a page a user asked aboutPerplexity says this fetcher generally ignores robots.txt
Google-ExtendedGoogleA control token, not a crawlerOpts out of Gemini training and grounding. Google says it doesn't affect Google Search
Applebot-ExtendedAppleA control token, not a crawlerOpts out of Apple's AI model training
CCBotCommon CrawlOpen web archive many AI models are trained onYour pages are left out of future Common Crawl archives

Sources: OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl. The robots.txt guide explains how to write the rules.

What Visited My Sites

I counted requests in the raw access logs of three of my sites, ivig.dev, mouse-jiggler.org and a product blog, from 1 to 12 November 2026. Then I checked each request's IP address against the address lists each company publishes.

20%
of 522 requests calling themselves GPTBot came from OpenAI's published addresses
My server logs, 1–12 Nov 2026
34%
of 1,947 requests calling themselves Googlebot came from Google's published addresses
My server logs, 1–12 Nov 2026
Requests by claimed crawler, three sites, 1 to 12 November 2026
Claimed crawlerRequestsFrom the published addresses
GPTBot522109 (20%)
ChatGPT-User612169 (27%)
OAI-SearchBot816365 (44%)
PerplexityBot776376 (48%)
Googlebot1,947667 (34%)
Bingbot1,4991,014 (67%)

I looked up the addresses that failed the check. The busiest ones were rented Google Cloud servers, with host names ending in googleusercontent.com, not Google's or OpenAI's crawlers. The same addresses claimed to be Googlebot and GPTBot. They were scrapers pretending to be well-known bots.

Two limits on these numbers. The address lists are today's, so a few genuine requests from addresses a company has since retired may count as unverified. And ClaudeBot and Perplexity-User couldn't be checked this way: Anthropic's help page doesn't list crawler addresses, and Perplexity's list for Perplexity-User held only ten. On ivig.dev in October, requests claiming to be ClaudeBot (1,393) outnumbered those claiming to be Googlebot (953), but I can't say how many of either were genuine.

How to Tell a Real Crawler From a Fake

The user agent is just text, and anyone can send it. Check the IP address instead:

  • OpenAI publishes address lists for each crawler: openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json.
  • Perplexity publishes perplexity.ai/perplexitybot.json and perplexity.ai/perplexity-user.json.
  • Google publishes its crawler addresses on its verifying Googlebot page. A reverse DNS lookup on a real Googlebot address ends in googlebot.com or google.com.
  • Bing publishes bing.com/toolbox/bingbot.json.
  • Common Crawl warns that crawlers falsely identify themselves as CCBot, and recommends verifying them.

Why it matters: robots.txt only works on crawlers that choose to obey it, and impostors don't. If fake bots are loading your server, block them by address or with your host's bot protection. Changing robots.txt won't stop them, and blocking by user agent would block the real crawlers too.

What to Allow and What to Block

Rules by goal
Your goalAllowBlock
Be cited in AI answers, and don't mind trainingEverything (the default)Nothing
Be cited in AI answers, but stay out of trainingOAI-SearchBot, Claude-SearchBot, PerplexityBot, and the user fetchersGPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot
Keep content away from all AI productsGooglebot and Bingbot for normal searchAll AI tokens. User fetchers may still visit, because some ignore robots.txt

For most businesses that want to be found, I'd use the first or second row. Blocking the search crawlers removes you from the answers your customers may be reading. My own sites allow everything.

The second row as robots.txt rules:

robots.txt: stay in AI search, opt out of training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

OpenAI says robots.txt changes take about 24 hours to affect search. Blocking a crawler doesn't remove content already collected. If an AI crawler needs your content, it also needs to see it in the HTML: most AI crawlers don't run JavaScript. For what else helps you get cited, see how to rank in ChatGPT.

Check Your Own Logs

Most hosts keep raw access logs. In cPanel, look for Raw Access in the Metrics section. This command counts requests per claimed AI crawler in a log file:

Terminal: count AI crawler requests in an access log
grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Googlebot|bingbot' access.log \
  | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn

For a compressed monthly log, start with zcat access-log.gz | instead. Treat the result as an upper limit until you've checked the addresses against the published lists.

Want to Know What AI Tools See on Your Site?

My AI search audit ($255, 2-day delivery) checks whether AI crawlers can reach and read your pages, tests how AI tools describe your business, and gives you a prioritized list of fixes. For a first look, request a free website review.

Request a free website review

Frequently Asked Questions

GPTBot is OpenAI's web crawler for collecting content that may be used to train its AI models. It obeys robots.txt. Blocking it opts you out of training but, according to OpenAI, doesn't remove you from ChatGPT search, which uses OAI-SearchBot.
ClaudeBot is Anthropic's crawler for collecting content that could be used to train Claude models. Blocking it in robots.txt signals that your future content should be left out of training. Anthropic uses separate agents, Claude-SearchBot and Claude-User, for search and for pages users ask about.
Only if you don't want your content used for training. It won't affect ChatGPT search, which uses OAI-SearchBot, or Google Search. Many businesses that want to be found leave it allowed.
Google says no. Google-Extended only controls whether content Google crawls can be used for Gemini training and grounding. It doesn't change how Googlebot crawls or ranks your site.
Many of those requests are fakes. In my logs, only 20% of requests claiming to be GPTBot and 34% claiming to be Googlebot came from the companies' published addresses. Scrapers borrow well-known bot names to avoid being blocked.

Related Guides

Milan Georgijevic iviG
Technical SEO Consultant & Developer

I'm a developer who moved into technical SEO. I trace search problems through crawling, rendering, indexing, templates, content and performance, then implement the fixes instead of handing over a generic export. Request a free website review and I'll review your site personally.

All SEO Guides