Robots.txt is a plain text file at the root of your site that tells crawlers which URLs they're allowed to fetch. Google's documentation defines its job narrowly: managing crawler traffic, "mainly to avoid overloading your site with requests". It controls crawling, not indexing, and nearly every expensive robots.txt mistake comes from confusing those two jobs. This guide covers how the file works, what belongs in it, what never does, and how to handle the new generation of AI crawlers.

I have a personal stake in this topic. The site you're reading launched with Disallow: / in its robots.txt, a leftover from pre-launch, and sat invisible to every search engine until I caught it. One line. That's all it takes, in either direction, which is why this file deserves fifteen minutes of actual understanding instead of a copied snippet from a forum.

500 KiB
The most robots.txt Google will read. Rules beyond that limit simply get ignored.
Google Search Central
24 h
How long Google may cache your robots.txt. Fixes aren't instant, so don't panic-edit twice.
Google Search Central
28 yrs
From Martijn Koster's 1994 invention to a formal internet standard, RFC 9309, in 2022
RFC Editor

What Robots.txt Actually Is

One file, one location: https://yourdomain.com/robots.txt. Crawlers request it before crawling anything else on the host, read the rules that apply to their user agent, and, if they're well-behaved, respect them. A complete, healthy robots.txt for a typical small site is three lines:

robots.txt
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

That says: every crawler, nothing blocked, and here's the sitemap. An empty Disallow means "allow everything". This site runs exactly that file, and you can read it live. Most small sites never need more.

Worth knowing where the rules come from: the protocol was invented by Martijn Koster in 1994 and ran on convention for decades before becoming a formal standard, RFC 9309, in September 2022. That history explains its quirks: it's simple, it's voluntary, and different crawlers interpreted it differently for years.

The Distinction That Prevents Disasters: Crawling vs Indexing

Robots.txt answers one question: may a crawler fetch this URL? It says nothing about whether the URL appears in search results. Google's robots.txt introduction is blunt about it: robots.txt "is not a mechanism for keeping a web page out of Google."

It gets more counterintuitive. A page blocked by robots.txt can still get indexed. If other sites link to the blocked URL, Google may index the address itself, showing it in results with no description, because it knows the page exists without being allowed to read it. And the trap inside the trap: a noindex tag on a robots-blocked page does nothing, because Google can't crawl the page to see the tag.

The rule to memorize: to keep a page out of Google's results, let it be crawled and give it a noindex tag (covered in my meta tags guide), or put it behind a password. To reduce pointless crawling of URLs that were never going to rank, use robots.txt. Wrong tool, wrong outcome, every time.

How Google Reads Your File

A handful of behaviors, all documented in Google's robots.txt specification, explain most of the "why is it ignoring my rules" confusion:

  • One file per host, protocol and port. The rules in https://example.com/robots.txt don't cover blog.example.com or the http version. Every subdomain needs its own file.
  • Paths are case-sensitive. Disallow: /Private/ doesn't block /private/. Field names like User-agent don't care about case; path values do.
  • It's cached up to 24 hours. Fixing the file doesn't take effect the moment you save. Give it a day before judging.
  • A missing file means "crawl everything". 4xx responses for robots.txt are treated as if no file exists. Deleting a broken robots.txt is a valid emergency fix.
  • A 5xx error freezes crawling. If your server errors when serving robots.txt, Google initially stops crawling the whole site for 12 hours, then falls back to a cached copy for up to 30 days. A misconfigured server rule on this one file can quietly pause your entire crawl.
  • 500 KiB is the ceiling. Content past that gets ignored. If your robots.txt is anywhere near it, something upstream has gone very wrong.

The Syntax That Matters

Google supports exactly four fields. Everything else in your file is decoration:

DirectiveWhat it doesGoogle's treatment
User-agentNames the crawler the following rules apply to. * matches allSupported. Most specific matching group wins
DisallowPath prefix the crawler may not fetchSupported. Empty value allows everything
AllowException that re-opens a path inside a disallowed oneSupported. More specific rule wins over shorter one
SitemapAbsolute URL of your sitemap, any number of linesSupported. Works for every crawler that reads the file
Crawl-delaySeconds between requests, in theoryNot supported. Google ignores it entirely

Two wildcards make paths flexible: * matches any run of characters, and $ pins the end of a URL. A few worked examples:

Pattern examples
# Block internal search results pages
Disallow: /search

# Block every URL containing a query string
Disallow: /*?

# Block PDFs only, wherever they live
Disallow: /*.pdf$

# Block a folder but allow one file inside it
Disallow: /assets/
Allow: /assets/logo.png

Remember these are prefix matches: Disallow: /search blocks /search, /search/anything and /searchable-page too. When a rule catches more than you intended, that last behavior is usually why.

What to Block, and What to Leave Alone

The test for every Disallow line: does crawling this URL waste the crawler's time without any chance of useful indexing? If yes, block it. If you're blocking because you don't want the page seen, you want a different tool.

Sensible to block

  • Internal search result pages
  • Faceted filter and sort URL explosions
  • Cart, checkout and account paths
  • Infinite spaces: calendars, session IDs, print views
  • Staging sites, as a backup behind real authentication

The CSS and JavaScript one deserves its sentence: Google renders pages like a browser, and if robots.txt blocks the files that build the layout, Google sees a broken page and judges it accordingly. Old WordPress advice to block /wp-includes/ aged badly for exactly this reason.

The AI Crawlers: New Tokens, Same File

Robots.txt found a second life as the opt-out switch for AI training. The major AI companies publish user agent tokens and state they respect the file. The ones that matter in practice:

TokenCompanyBlocking it affects
GPTBotOpenAITraining data collection for GPT models
OAI-SearchBotOpenAIChatGPT search results and citations of your site
ClaudeBotAnthropicTraining data collection for Claude models
PerplexityBotPerplexityPerplexity's search index and citations
Google-ExtendedGoogleGemini training and grounding. Not Google Search

Two facts before you block anything. First, per Google's crawler documentation, Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". AI Overviews ride normal Google indexing, so blocking Google-Extended won't pull you out of them. Second, compliance is voluntary: reputable companies honor these rules, scrapers never did.

Whether to block is a business decision, not a technical one. For a business site like mine, AI assistants citing my content is free distribution, so I block nothing. A publisher selling the content itself may reason differently. Decide from your model, then write the three lines.

Which Tool Do You Actually Need?

Say what you're trying to accomplish, and get the right mechanism. This runs entirely in your browser.

What are you actually trying to do?

The Mistakes I Keep Finding on Audits

  1. The staging block that shipped. Disallow: / goes live with the redesign, and traffic bleeds out silently. It happened to this very site at launch. Make "check robots.txt" a deployment step, not an incident response.
  2. Robots.txt as a privacy tool. The file is publicly readable. Listing /admin/ or /internal-reports/ hands every curious visitor a map of what you consider sensitive, while blocking only the bots polite enough to ask.
  3. Blocking a page and noindexing it. The block stops Google from ever reading the noindex, so the URL can linger in results anyway. Pick one mechanism per goal.
  4. Blocking rendering assets. CSS and JS behind a Disallow turn your pages into broken skeletons in Google's eyes, and mobile-friendliness checks fail for no visible reason.
  5. Forgetting subdomains are separate. The main site's file doesn't cover blog., shop. or staging. subdomains. Each host answers for itself.
  6. Overreaching prefixes. Disallow: /page also kills /pages/pricing. Prefix matching plus wildcards means every rule deserves a test against real URLs before it ships.
  7. The file that errors. A server rule that 5xxes on robots.txt can freeze crawling of the whole site for days. If nothing else, make sure this one URL always returns 200 or 404, never 500.

Testing Your File

Three checks, thirty seconds each:

  1. Fetch it like a crawler would. Open yourdomain.com/robots.txt in an incognito window, or from a terminal:
Terminal check
curl -sI https://example.com/robots.txt  # status must be 200 (or 404 if you have no file)
curl -s https://example.com/robots.txt   # read what crawlers actually receive
  1. Check what Google has. Search Console's robots.txt report (under Settings, Crawling) shows the fetched file, when it was fetched, and parse errors. This is Google's copy, cache and all, not your server's.
  2. Test specific URLs. Run your important pages through URL Inspection. "Blocked by robots.txt" on a money page is the finding that pays for the whole exercise. My indexing guide covers what to do with every verdict.

Copy-Paste Templates

Three starting points covering most real situations. Adjust paths to your site before shipping:

1. Safe default for most sites
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml
2. Typical site with junk URLs to contain
User-agent: *
Disallow: /search
Disallow: /cart
Disallow: /checkout
Disallow: /*?sort=
Disallow: /*?filter=

Sitemap: https://example.com/sitemap.xml
3. Allow search engines, opt out of AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

The Robots.txt Checklist

Run this on your live file today

  • File lives at the root and returns 200 (or intentionally 404)
  • No Disallow: / unless you genuinely mean to vanish
  • CSS, JS and images needed for rendering are not blocked
  • Nothing sensitive is "protected" by a Disallow line
  • No page is both robots-blocked and noindexed
  • Every rule tested against real URLs, prefixes checked
  • Sitemap line present with the absolute URL
  • Every subdomain has its own correct file
  • AI crawler policy decided deliberately, not by default
  • Search Console's robots.txt report shows no parse errors

Not Sure What Your Robots.txt Is Costing You?

A wrong line in this file is invisible until the traffic graph tells you. I check robots.txt, indexing, canonicals and the whole crawl path in my free audit, personally, and tell you what I'd fix first.

Get My Free Audit

Robots.txt: FAQ

No. Google's docs say it directly: robots.txt is not a mechanism for keeping a page out of Google. A blocked URL can still be indexed from external links, just without its content. To remove a page from results, allow crawling and use noindex, or password-protect it.
Not strictly. No file means crawlers may fetch everything, which suits many small sites fine. You still want one for the Sitemap line and to contain crawl traps like internal search. A missing file is harmless; a wrong one isn't.
No. Compliance is voluntary. Google, Bing and the major AI companies document and honor it; scrapers and malicious bots ignore it entirely. Treat it as a signpost for well-behaved crawlers, and handle hostile bots at the server or firewall level.
No. Google's documentation states fields like crawl-delay aren't supported. If Googlebot hits your server too hard, the levers are your server capacity and Search Console's crawl settings, not that line. Bing does read it, for what it's worth.
The opposite happens. The file is publicly readable, so listing /admin or /reports in it publishes a map of what you're hiding, while stopping only the polite bots. Private content needs authentication. Robots.txt is crawl management, not security.
Add Disallow rules for their tokens: GPTBot, ClaudeBot, PerplexityBot, and Google-Extended for Gemini training. Compliance is voluntary, and blocking Google-Extended doesn't affect Google Search or AI Overviews. Decide whether AI visibility earns or costs you business before blanket-blocking; for most service businesses, being cited is free marketing.
Milan Georgijevic, technical SEO consultant
Milan Georgijevic
Technical SEO Consultant & Developer

I'm a developer who moved into technical SEO. I audit sites, then implement the fixes myself: indexing, Core Web Vitals, structured data, rendering. Robots.txt is the second file I open on every audit, right after the traffic graph. Request a free technical audit and I'll take a look personally.

All SEO Guides