How to Use Cloudflare AI Crawl Control for Search Visibility

To use Cloudflare AI Crawl Control for search visibility, start with the Crawlers tab to see which AI bots actually hit your site. Set Allow on the search and user-fetch crawlers you want answering questions about you (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User), and Block only the ones you’ve decided against. Then open your AI bot policies, because a Training block also catches mixed-purpose crawlers, and Help Net Security names Googlebot, Applebot and Bingbot among them. Make robots.txt say the same thing as the dashboard. Finally, request a real page as each crawler and confirm it gets a 200.

Here’s the usual situation. You run an agency or a small practice, and your site sits behind Cloudflare. A year ago someone on the team clicked “block AI bots” because it sounded safe. Now a prospect says ChatGPT can’t find your service pages, and nobody knows which setting did it.

I’m Satya Phanindra Reddy, founder of Magic Teams AI. This post covers the order we use, what each Cloudflare control does, and the checks we ran on October 19, 2026 on this blog, which is served through Cloudflare.

What does Cloudflare AI Crawl Control actually do?

AI Crawl Control (formerly AI Audit) shows which AI services request your content and lets you allow or block each one. Cloudflare lists it as available on all plans.

You’ll use four parts of it:

• Crawlers tab. One row per AI crawler, with its operator, request counts and an action you can set (Cloudflare docs).
• Metrics tab. Requests over time, status codes and popular paths. Referral data is paid plans only (docs).
• Directives tab. Whether your robots.txt is reachable, and which crawlers requested paths it disallows (docs).
• Block response settings. On paid plans, you choose whether blocked crawlers get 403 Forbidden or 402 Payment Required.

Two details matter for visibility. When you block a crawler, Cloudflare says it “creates or updates a WAF custom rule on your zone to enforce that block.” So a bot can be blocked even when robots.txt says Allow.

The docs also warn that unsuccessful requests “may come from any rule or response error, not just the block action in AI Crawl Control.” So a bot marked Allow can still get stopped by a different security rule.

One more limit. On the Free plan, Cloudflare identifies AI crawlers by their user agent strings. Paid plans add Bot Management detection IDs, which catch more.

Which AI crawlers should you allow for search visibility?

Most AI companies run separate bots for search indexing, live user requests and model training. If you want AI answers to cite you, you need the first two. Training is a separate business decision.

Operator	Search / index	User-triggered fetch	Training	Source
OpenAI	OAI-SearchBot	ChatGPT-User	GPTBot	OpenAI bots
Anthropic	Claude-SearchBot	Claude-User	ClaudeBot	Anthropic help
Perplexity	PerplexityBot	Perplexity-User	Not listed	Perplexity bots
Google	Googlebot	See Google’s fetcher list	Google-Extended (a robots.txt token, not a separate bot)	Google crawlers

What the vendors say about these bots:

• OpenAI says sites that opt out of OAI-SearchBot won’t be shown in ChatGPT search answers, though they can still appear as navigational links. It also says robots.txt rules “may not apply” to ChatGPT-User, because a person triggers those visits.
• Anthropic says blocking Claude-SearchBot prevents indexing for search, and blocking Claude-User stops Claude fetching your pages for user questions. Both “may reduce your site’s visibility.”
• Perplexity says Perplexity-User “generally ignores robots.txt rules” since a user requested the fetch. If you really need it out, robots.txt won’t do it. An edge block will.
• Google says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”

In practice, “allow AI search bots” means this: Allow the search and user-fetch columns, then make a separate decision about training.

Why can a Training block hide you from Google and Bing?

This setting is the one most likely to hurt you without anyone noticing.

Cloudflare now offers separate AI bot policies for Search, Agent and Training, under Security Settings > Configure AI bot policies. Each can be set to Block on all pages, Block on pages with ads, or Allow (Cloudflare docs).

The catch is one sentence on that page: “Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy ‘Block AI bots’ option.” Cloudflare’s own page doesn’t name those crawlers. Help Net Security’s coverage does: “multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked, even when Search crawlers are allowed.”

The defaults moved too. Cloudflare says that from September 15, 2026, new domains block Training and Agent bots on pages that display ads, while Search stays allowed. The old single toggle is marked as deprecating.

That matters for ad-supported pages. The Agent category covers chat fetch bots, so on those pages ChatGPT-User or Claude-User requests can be refused by default.

How do robots.txt and AI Crawl Control fit together?

They do different jobs. robots.txt states your preference. AI Crawl Control and WAF rules enforce it at the edge.

Cloudflare says it plainly: robots.txt “expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level” (managed robots.txt docs). If you turn on managed robots.txt and you already have a file, Cloudflare prepends its own directives and Content Signals to it.

Here’s how a crawler request moves through those layers.

One rule keeps you out of trouble: the file and the dashboard should say the same thing. If robots.txt allows GPTBot but the Crawlers tab blocks it, the block wins.

If robots.txt disallows a bot but the dashboard allows it, you’re trusting that bot to comply. The Directives tab shows you which ones don’t. For file-level problems, see our guide to debugging robots.txt for AI crawlers.

How do you set it up, step by step?

This is the order we follow. Steps 1 to 3 change nothing, so do them before touching a setting.

1. Inventory. Open AI Crawl Control > Crawlers. Note which bots actually request your pages and what action each one has now.
2. Read the status codes. On the Metrics tab, look at the status code chart. A rise in 4xx responses for search bots usually means some rule is stopping them.
3. Check the policy page. Security Settings > Configure AI bot policies. Write down Search, Agent and Training, and whether the legacy “Block AI bots” toggle is on.
4. Decide per category. Search: Allow if you want AI search citations. Agent: Allow if you want assistants to fetch pages when users ask. Training is your call, but a block also reaches mixed-purpose crawlers.
5. Set per-crawler actions. In the Crawlers tab, Allow the search and user-fetch bots and Block only the ones you’ve ruled out. For path exceptions, extend the WAF rule Cloudflare created.
6. Align robots.txt. Write explicit groups for the bots you care about and keep your sitemap line. Decide on purpose whether managed robots.txt prepends to your file.
7. Verify. Fetch robots.txt and a few key pages, then watch the Directives and Metrics tabs for the next week.

A quick caveat: WAF and bot rules often protect logins, forms and client portals too. If someone else manages your security setup, check with them before loosening anything.

Most of step 4 comes down to one question.

Worked example: what we checked on this blog

Here’s the review we ran on blog.magicteams.ai on October 19, 2026. It’s a small B2B content site behind Cloudflare, and we want it findable in both traditional and AI search.

robots.txt. The live file has explicit Allow groups for OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended and Bingbot. It disallows /admin and /dashboard for everyone else and points to the sitemap index.

It also carries a Content Signals line: search=yes, ai-input=yes, ai-train=no, use=reference. A comment above it says the signal is advisory and Cloudflare is the enforcement layer.

There’s a tension in that file worth naming. GPTBot and ClaudeBot may crawl, yet the signal says ai-train=no. We did that on purpose. Allow controls access, and the signal says how we’d like the content used. That only works if crawlers honor Content Signals, so we treat it as a preference, not a guarantee.

Responses. We requested a live article with user agents for OAI-SearchBot, GPTBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Googlebot. Every one returned 200.

That test has a limit. A curl request with a bot’s name in the user agent isn’t a verified bot, and Cloudflare can treat real crawler traffic differently. It proves no rule blocks those user agents on that path. It doesn’t replace the Crawlers tab.

What we didn’t check. This review left out the zone’s dashboard: the Crawlers tab actions, the Search, Agent and Training policies, and the legacy toggle. Those come next on the checklist below, and they’re where a Training block would show up.

What’s on the Magic Teams Cloudflare review checklist?

This is the list we work through. We built it from the Cloudflare docs above plus the mistake in that callout. We ran the robots.txt and response items on this blog. The dashboard items get checked in the zone itself.

Cloudflare AI crawler review

• Crawlers tab: list every AI bot with requests and its current action
• AI bot policies: record Search, Agent and Training settings and the legacy toggle
• Confirm a Training block isn't catching Googlebot, Bingbot or Applebot
• Check whether Agent is blocked on pages with ads
• Search and user-fetch bots set to Allow if AI citations matter
• robots.txt returns 200, matches dashboard actions, includes sitemap
• Managed robots.txt on or off on purpose, with prepended rules reviewed
• Fetch three sitemap URLs as each key bot and confirm 200
• Directives tab: review violations a week after any change
• Write down who owns the Cloudflare settings and when they were last reviewed

The last item sounds minor. It’s the one that keeps the “someone clicked a toggle a year ago” problem from coming back.

FAQ

Does allowing AI crawlers guarantee citations in ChatGPT or Perplexity?

No. Letting crawlers in only makes a citation possible. Whether you actually get cited depends on the content. Start with what answer engine optimization means for B2B and how to structure a page so AI search can cite it.

Is AI Crawl Control free?

Cloudflare lists it on all plans. On Free, detection relies on user agent strings. Custom block responses and referral data need a paid plan, and pay per crawl is in private beta.

Should I block training crawlers?

That’s more of a business and legal call than a technical one. If client confidentiality or licensing terms are involved, talk to your lawyer. Our AI data privacy guide for agencies covers the operational side. If you do block, check what happens to mixed-purpose crawlers before saving.

Can I just rely on robots.txt?

Only for crawlers that honor it. Cloudflare says robots.txt doesn’t technically prevent access, and Perplexity says its user fetcher generally ignores it. When a block has to hold, use the dashboard and WAF.

How often should I review these settings?

Review them after any Cloudflare policy change, after a site migration, and on a regular schedule. The September 2026 default change shows that a new zone can behave differently from an old one without anyone on your team touching a setting.

Where does this fit in a bigger setup?

Cloudflare settings are one small piece of running a business that’s easy to find and easy to operate. In a Magic Teams AIOS install, crawler access sits next to the content, data handling and ownership rules that keep a setting correct after we leave.

We can help with the technical setup, but nobody can promise citations. If you’d like a second pair of eyes on how your site and operations fit together, book a conversation and we’ll start there.