How to Debug robots.txt for AI Crawlers

To debug robots.txt for AI crawlers, check five things in order: the live HTTP response for /robots.txt, which user-agent group each AI bot actually lands in, whether the sitemap URL resolves, whether “private” paths are protected by something stronger than robots.txt, and whether the layer that enforces your policy (CDN, firewall, bot manager) matches what the file says. Most bugs live in the first two. The file in your repo isn’t always the file crawlers see, and a bot that matches a named group ignores everything you wrote under User-agent: *.

Here’s the usual situation. You run an agency, someone asks “are we blocking ChatGPT?”, and three people give three answers. One reads the repo, one reads a plugin setting, one checks Cloudflare.

This post gives you one order of checks, the rules behind each, and a worked example from our own blog, bug included. I’m Satya Phanindra Reddy, founder of Magic Teams AI, and I ran the audit below on the file that serves this site.

Why does robots.txt behave differently for AI crawlers?

The file format is the same. What’s different is the number of bots and what each one is for.

Most AI companies now run several crawlers with separate jobs: one for search, one for model training, one that fetches a page when a user asks. Each reads robots.txt under its own name. Block the wrong one and you either opt out of AI search by accident or think you’ve opted out of training when you haven’t.

Here’s what the operators’ own documentation says, checked on October 18, 2026.

Token	Operator	What it does	Does robots.txt control it?
OAI-SearchBot	OpenAI	Surfaces sites in ChatGPT search	Yes, per OpenAI’s bot docs
GPTBot	OpenAI	Crawls content that may be used to train models	Yes, disallowing it is the training opt-out
ChatGPT-User	OpenAI	User-initiated actions in ChatGPT	OpenAI says robots.txt rules “may not apply”
ClaudeBot	Anthropic	Collects content for model training	Yes, per Anthropic’s help article
Claude-SearchBot	Anthropic	Indexes content for Claude’s search answers	Yes
Claude-User	Anthropic	Fetches pages when a Claude user asks	Yes, Anthropic says it lets site owners control which sites it can access
PerplexityBot	Perplexity	Surfaces and links sites in Perplexity search	Yes, per Perplexity’s bot docs
Perplexity-User	Perplexity	User-initiated fetches	Perplexity says it “generally ignores robots.txt rules”
Google-Extended	Google	Control token for Gemini training and grounding	Yes, but it isn’t a separate crawler

That last row trips people up. Google says Google-Extended doesn’t have a separate HTTP request user agent string. You won’t see it in your logs. It only changes how Google may use what its existing crawlers already fetched.

What’s the right order to debug in?

Follow the order a crawler follows when it reads the file, then add the one check that sits outside the file. We call it the Five-Point Robots Check: Response, Groups, Sitemap, Private paths, Enforcement.

It isn’t a proprietary method. It’s the processing order from RFC 9309 turned into a checklist, plus an enforcement step. We added that step because “the file says allow” and “the bot gets blocked” can both be true at once.

The flow below shows the sequence and what you’re looking for at each step.

Step 1: What does the live robots.txt response actually say?

Start with the URL, not the repo. CDNs, CMS plugins, and bot managers can rewrite or replace the file on the way out.

curl -sI https://example.com/robots.txt
curl -s  https://example.com/robots.txt
curl -s -A "GPTBot" -o /dev/null -w "%{http_code}\n" https://example.com/robots.txt

Read the status code first, because it changes the meaning of everything else:

• 200: the file is parsed as written.
• 4xx (except 429): Google treats it as if no robots.txt exists, so nothing is restricted. RFC 9309 says crawlers may access any resource.
• 5xx or unreachable: RFC 9309 says crawlers must assume complete disallow. Google stops crawling for the first 12 hours, then uses the last good copy for up to 30 days while it keeps retrying.

The third command matters more than it looks. If your firewall returns 403 to anything calling itself GPTBot, that bot may never read your rules at all. Anthropic warns that blocking its IP addresses impedes its ability to read your robots.txt and may not guarantee an opt-out.

A spoofed user agent only tests your own firewall rules, not what the real bot sees. It’s still the quickest way to catch a blanket block.

Also check two quieter things. The file should be plain text, and Google ignores content past 500 KiB. If you’re on Cloudflare with managed robots.txt switched on, Cloudflare prepends its own block to yours, so the top of the live file won’t match your source.

Step 2: Which user-agent group does each AI bot fall into?

This is where most AI-crawler bugs hide. A crawler obeys the one most specific group that matches its name and ignores the rest. Google states it directly: user-agent-specific groups and the global * group are not combined.

So if you write a friendly User-agent: GPTBot / Allow: / block, GPTBot stops reading your * group. Every Disallow you put there no longer applies to it.

A few more rules to check against:

• User-agent names are case-insensitive. gptbot and GPTBot match the same crawler.
• Paths are case-sensitive. Disallow: /Admin doesn’t block /admin.
• The most specific (longest) matching path wins. When rules still conflict, Google uses the least restrictive one, the Allow.
• Repeated groups for the same bot get merged. Two User-agent: ClaudeBot blocks in different parts of the file count as one.
• Order doesn’t matter. Moving a group higher won’t give it priority.

The fastest test is a small grid. For each bot in the table above, write down which group it matches and the final rule for three URLs: your homepage, one blog post, and one path you meant to keep out. If you can’t answer from the file alone, the file is too clever.

Step 3: Does the sitemap line point to a real, crawlable sitemap?

The Sitemap: line isn’t part of any user-agent group. It applies to everyone.

Google requires a fully qualified URL, including protocol and host. A relative path like /sitemap.xml is a common error. You can list more than one sitemap, and it can live on another host.

Then fetch it:

curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/sitemap-index.xml

You want a 200 and an XML content type. A sitemap that redirects to a login page, or one that lists URLs you’ve disallowed, sends mixed signals worth fixing.

Step 4: Are your private paths protected by something other than robots.txt?

Robots.txt is a public file. Anyone can read it, which means every path you list there is advertised.

RFC 9309 says its rules “are not a form of access authorization.” Google adds that robots.txt isn’t a mechanism for keeping a page out of Google: a blocked URL can still be indexed if other pages link to it. To keep something out of results, use noindex or put it behind a password.

So for each path in your Disallow lines, ask what actually stops a stranger. Login? Signed URLs? noindex? If the answer is “the robots file,” that’s the bug.

A plain-language caveat: this is a security question, not an SEO one. If a disallowed path holds client files, financial records, or anything regulated, have whoever owns your security or compliance confirm the real access controls.

Step 5: Does the enforcement layer match the policy?

Robots.txt asks. It doesn’t enforce. Cloudflare’s own docs say “robots.txt compliance is voluntary”, and two of the user-triggered fetchers in the table above say robots.txt may not apply to them.

If you need a bot actually stopped, that happens at your CDN, firewall, or bot manager. Cloudflare’s AI Crawl Control lets you allow or block each crawler, and its robots.txt tracking flags crawlers that violate your directives.

The debugging question is whether the two layers agree. Two failure patterns show up:

• File allows, firewall blocks. You think you’re visible in AI search, but the search bot gets a 403 or a challenge page.
• File blocks, nothing enforces. You think you’ve opted out, but a fetcher that doesn’t follow robots.txt still pulls pages.

Neither shows up by reading the file. Put the CDN settings and your server logs side by side.

Which check fixes which symptom?

When someone reports a specific problem, use this map instead of running all five checks from scratch.

After any fix, give it time. OpenAI says it can take about 24 hours for its search systems to adjust, Perplexity says up to 24 hours, and Google generally caches the file for up to 24 hours. For Google, the Search Console robots.txt report shows the version it fetched and lets you request a recrawl in urgent cases.

Worked example: what did the check find on our own blog?

We ran the Five-Point Robots Check on blog.magicteams.ai/robots.txt on October 18, 2026. This is our real, public file. Here’s its shape, trimmed:

Content-signal: search=yes, ai-input=yes, ai-train=no, use=reference

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

(…same pattern for OAI-SearchBot, ChatGPT-User, Claude-SearchBot,
Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Bingbot)

User-agent: *
Allow: /
Disallow: /admin
Disallow: /dashboard

Sitemap: https://blog.magicteams.ai/sitemap-index.xml

Step 1, response: pass. The live file returns 200 as text/plain and matches the source in our repo. There’s no prepended block, so Cloudflare’s managed robots.txt isn’t rewriting it.

Step 2, groups: fail. Every bot we named has its own group with only Allow: /. Because named groups aren’t combined with *, none of those ten tokens (nine AI ones plus Bingbot) carries the /admin or /dashboard exclusions. Google-Extended never fetches pages itself, so the real gap is for the other nine, which actually crawl.

We wrote the named groups to be welcoming. In doing so, we removed the exclusions for exactly the bots we cared most about.

Step 3, sitemap: pass. The URL is absolute and returns 200 with application/xml.

Step 4, private paths: pass, for the right reason. /admin is our CMS login screen, and editing requires signing in with GitHub. The robots.txt line was never what protected it.

Step 5, enforcement: open item. The file can’t tell us what our Cloudflare AI Crawl Control settings are. That needs a dashboard check against the policy.

The checks also surfaced a policy mismatch. The Content-signal line says ai-train=no, but the file explicitly allows GPTBot, ClaudeBot, and Google-Extended, the training-related tokens. A comment in our source says allowing training crawlers was deliberate. So the file says two different things, and a reader can’t tell which we mean.

Here’s the before and after.

Before: separate named groups

• GPTBot group: Allow / only
• Disallow /admin sits under * only
• Ten named bots skip the exclusions
• Content-signal says ai-train=no while training bots are allowed

→

After: one shared group

• All named bots and * share one rule set
• Disallow /admin and /dashboard apply to everyone
• Training policy decided once, written once
• CDN settings checked against the file

The simplest fix is one group with several User-agent lines, which RFC 9309 allows:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Bingbot
User-agent: *
Allow: /
Disallow: /admin
Disallow: /dashboard

The rules now live in one place, so they can’t drift between groups. Listing the bots by name still documents who you expect.

Then settle the training question. If you want training crawlers in, change ai-train=no to yes. If you want them out, move GPTBot, ClaudeBot, and Google-Extended into their own group with Disallow: /. Either is defensible. Saying both isn’t.

Frequently asked questions

If I block GPTBot, do I disappear from ChatGPT search?

Not according to OpenAI. Search uses OAI-SearchBot, and training uses GPTBot. OpenAI’s docs treat them as separate controls, so you can opt out of one and keep the other.

Does blocking Google-Extended hurt my Google rankings?

Google says no. Google-Extended doesn’t affect a site’s inclusion in Google Search and isn’t used as a ranking signal. It controls use in Gemini training and grounding.

Can I test robots.txt for AI bots the way I test it for Google?

Partly. Search Console only reports on Google’s fetches. For AI bots, you’re left with curl using their user-agent strings, your server or CDN logs, and the IP lists the operators publish so you can tell real bots from impostors.

Do subdomains need their own robots.txt?

Yes. Robots.txt applies per host. Anthropic specifically says to add the rules for every subdomain you want to opt out.

Is robots.txt enough to stop my content being used for AI training?

It’s the documented opt-out for the crawlers that honor it, but it’s a request, not a contract or a technical barrier. If you have legal questions about how your content is used, talk to a lawyer who works on IP and data rights.

Where does this fit in a wider AI search setup?

Robots.txt is the front door of answer engine optimization for B2B: if the right bots can’t read your pages, nothing downstream matters. It’s also a file that drifts. Someone adds a bot, a plugin changes a setting, a CDN rule gets copied from another site.

That’s why we’d put the Five-Point Robots Check inside a recurring review step, as described in how to build an AI search content system, rather than running it once. If you’re weighing whether that review is worth the time, measure it like any other automation with how to measure ROI on AI automation.

At Magic Teams, we set up connected, reviewable workflows like this one for founders whose site, content, and checks have drifted into separate jobs. If that sounds like your setup, a fit call is a quick way to find out whether it’s the right problem to solve now.