Magic Teams
October 21, 2026

Robots.txt for AI Search Bots: What Should You Allow?

By Phanindra Reddy · AEO & AI Search

Policy matrix mapping bot type (search, assistant, training, preview, private) to purpose, default action and enforcement layer
Photo: Magic Teams AI / generated in the build

Allow the AI crawlers behind the search results and answers you want to appear in: OAI-SearchBot, Claude-SearchBot, PerplexityBot, plus Googlebot and Bingbot. Decide separately about training controls like GPTBot, ClaudeBot and Google-Extended. Keep private and unfinished routes behind a login, because a Disallow line won’t keep anyone out.

Robots.txt is a polite request. Reputable bots honor it, and some user-triggered fetchers don’t. Anything that must stay out needs a technical block at the edge or on the server.

Here’s a common setup. You run an agency or a small advisory practice. Someone read that AI companies are scraping everything, so the site got a blanket Disallow: / for every AI bot they could name. Six months later a prospect asks ChatGPT about your niche and you’re not in the answer, because the search crawler got turned away along with the training one.

This post shows you how to decide one bot type at a time instead of all at once.

Why isn’t “allow or block AI bots” one decision?

“AI bot” covers at least four different jobs. Each major vendor now splits its crawlers by purpose, and blocking one doesn’t block the others.

So you’re not answering “AI or not.” You’re answering “which of these jobs do I want done with my pages?”

What does the Magic Teams bot policy matrix look like?

We sort bot traffic into five classes: search, assistant, training, preview and private. We built the matrix by reading each vendor’s crawler documentation and grouping bots by what they do with your page, not by which company runs them. Use it as a starting default, then adjust it to your business.

Bot class Purpose Example user agents or tokens Default action Enforcement layer
Search Indexes pages so answers can cite and link them Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot (Applebot is mixed-purpose) Allow on public pages robots.txt, with sitemap listed
Assistant Fetches a page live because a user asked ChatGPT-User, Claude-User, Perplexity-User Allow on public pages robots.txt may be skipped; edge rules if a block must hold
Training Collects content that may train future models GPTBot, ClaudeBot, Meta-ExternalAgent, plus the Google-Extended and Applebot-Extended tokens Business decision; write it down robots.txt for honest bots, Cloudflare block if it must hold
Preview Builds link cards when someone shares your URL facebookexternalhit, LinkedInBot, Slackbot Allow on pages you share Usually none needed
Private Anything hitting client portals, drafts, staging, admin Any bot, AI or not Deny Login or HTTP auth

Why split it like this? Because a single setting can’t do two jobs. One “block AI” switch can hide your service pages from AI search while a staging folder stays open to anything that ignores robots.txt. With separate classes, each decision has to be made out loud.

Here’s the same matrix as a question you can ask about any bot in your logs.

Is robots.txt actually enforced?

No. The standard itself, RFC 9309, says its rules “are not a form of access authorization.” It also warns that any path you list in robots.txt becomes publicly discoverable.

Google says the same thing in plainer words. Robots.txt can’t enforce crawler behavior, and a disallowed page can still get indexed if other sites link to it. To keep a page out of Google, it recommends noindex or password protection.

User-triggered fetchers are where this bites. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person started the request. Perplexity says its user fetcher generally ignores robots.txt. Meta says facebookexternalhit might bypass robots.txt during security or integrity checks.

That’s why the matrix has an enforcement column. There are three layers, and only the top one is a lock.

When should you use Cloudflare instead of robots.txt?

Use Cloudflare when a block has to hold. When you set a crawler to Block in AI Crawl Control, Cloudflare creates a WAF custom rule that returns a 403, or a 402 if you want to signal that the crawler must pay. On the Free plan, it identifies AI crawlers by user agent string, which is easier to fake than the bot detection on paid plans.

Cloudflare’s managed robots.txt is a separate feature. It writes Disallow lines and Content Signals (search, ai-input, ai-train) into your file for you. Cloudflare is blunt about the limit: robots.txt compliance is voluntary and doesn’t stop access at a technical level.

One trap to know before you click anything. Cloudflare’s AI bot policy page says that from September 15, 2026, “Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training.” Our Cloudflare AI Crawl Control walkthrough covers which settings to check so a training block doesn’t hide you from Googlebot or Bingbot.

What does a sensible robots.txt look like? A worked example

Take an illustrative 15-person agency site. It has public service pages and a blog, a /client-portal/ behind a login, a /drafts/ folder its CMS exposes, and a staging copy on a staging. subdomain. The owner wants to show up in AI search but hasn’t decided on training, so for now they opt out.

Before touching robots.txt, they fix the drafts. The folder moves behind the same login as the portal, and staging gets HTTP authentication. Then the matrix gives this file:

# Search and assistant: allowed on public pages
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
Disallow: /client-portal/

# Training: opted out for now (reviewed quarterly)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /client-portal/

Sitemap: https://example.com/sitemap.xml

Four choices in that file are easy to get wrong.

The private path repeats in every group. Under RFC 9309, a crawler follows the group that names it and ignores the * group. Give OAI-SearchBot its own group without the Disallow line and you’ve invited it into /client-portal/.

The Disallow line isn’t the protection. The login is. The portal path is already visible on the site’s login link, so listing it costs nothing, but you shouldn’t list paths you’d rather nobody discovered. Staging gets its own HTTP auth, because a robots.txt on staging is only a request too.

Don’t pair Disallow with noindex. If the owner had left /drafts/ public with a noindex tag and also disallowed it, Google never sees the noindex rule, and a linked draft could still show up in results. Pick one: auth (best), or noindex with crawling allowed.

The training opt-out has a cost and a review date. Blocking Google-Extended also opts out of grounding in Gemini Apps, so it isn’t free if Gemini visibility matters to you. The owner writes that trade-off down with a review date. If the opt-out ever has to hold against bots that ignore the file, they add a Cloudflare block instead of making robots.txt longer.

To check the live file after you deploy it, use our robots.txt debugging guide for AI crawlers.

Should you block AI training crawlers?

It’s a business and licensing call more than a technical one. Blocking honest training crawlers doesn’t remove you from Google Search or Apple search, per Google and Apple. It also doesn’t undo anything already collected.

Weigh three things. Do you sell your content or license it? Would you rather your public explanations shape what models say about your niche? And does the Gemini grounding trade-off above matter to your buyers?

For agencies and professional practices, public marketing pages usually aren’t the big risk. Client material ending up public at all is. If client contracts, confidentiality duties or professional conduct rules are involved, talk to your lawyer. This post isn’t legal advice.

When we set up an AIOS, we handle private routes and client material before any search or training setting, because a mistake there costs the most. Our guides on AI data privacy for agencies and whether it’s safe to put company data in ChatGPT cover the operational side.

What should you check before you change anything?

Run through this list before you flip a blanket block.

OpenAI says search changes can take about 24 hours to reflect a robots.txt update, so check again the next day.

FAQ

Not by itself. ChatGPT search uses OAI-SearchBot, a separate user agent. Block GPTBot to opt out of training and leave OAI-SearchBot allowed if you want to appear in answers.

Does blocking Google-Extended hurt my Google rankings?

Google says Google-Extended doesn’t affect inclusion in Google Search and isn’t a ranking signal. It does cover Gemini training and grounding in Gemini Apps and Vertex AI, so it’s not a zero-cost switch.

Can robots.txt keep my client portal private?

No. It’s a public file and a request. Use a login or HTTP authentication, and treat robots.txt as a hint to well-behaved crawlers.

Does allowing AI search bots guarantee I’ll be cited?

No. Allowing them only makes it possible. Whether an answer cites you depends on your content and on factors nobody outside those companies controls.

Should I block bots by IP address?

Be careful. Anthropic advises against blocking its IP addresses, because it can stop the bot from reading your robots.txt, which means the opt-out may not hold. Edge rules based on verified bot identity are usually cleaner.

Where does this fit?

Bot rules are a small part of a bigger question: what’s public, what’s private, and who owns that decision after setup. In a Magic Teams AIOS install, the bot policy sits next to the data handling rules for client material, so one toggle can’t quietly undo the other.

The crawler behavior described above comes from each vendor’s own documentation. The order of operations (private routes first, then search, then training) is how we work. Vendors change their bots often, so recheck their pages before you rely on any single rule.

Run the matrix before you flip a blanket block that also hides your useful pages. If you’d like a second pair of eyes on how your site and your client data are split, we’re happy to talk it through.

Written by Satya Phanindra Reddy, founder of Magic Teams AI, which installs local-first, human-in-the-loop AI operating systems for agencies and professional practices.