Cloudflare's AI Crawl Control Blocks ChatGPT on Sept 15

Q: Should I block CCBot and other training-only crawlers?

Yes, that's defensible. CCBot, Bytespider, anthropic-ai, and cohere-ai scrape content to train future models and give you no attribution, referral traffic, or citation back. Blocking them in robots.txt or Cloudflare's per-bot toggles costs nothing in AI search visibility.

I was auditing a client’s Cloudflare dashboard the other night before adding their site to Search Console. Nothing dramatic — a small business page on Cloudflare Pages, static Astro build, run by someone who bought the domain themselves and pushed the first deploy from a laptop. The kind of setup that exists by the million.

I opened the AI Crawl Control panel out of habit. And there, sitting on a screen almost no small-business owner has ever navigated to, was a radio button that read:

Mixed purpose crawlers will be blocked on September 15.

That default ships on every new zone and is quietly rolling out to existing ones. Mixed-purpose crawlers are the bots that do double duty: GPTBot (ChatGPT), ClaudeBot (Claude), and PerplexityBot (Perplexity) both train on your content and fetch your pages in real time to answer a user’s question. If the default stays selected, your site stops appearing in ChatGPT, Claude, and Perplexity answers starting September 15, 2026 — even though nothing about your SEO or your rankings changed. No email warns you. No dashboard banner flags it. Cloudflare sits in front of roughly a fifth of the web, so one default will quietly affect millions of sites. The fix takes two clicks: open Security → AI Crawl Control and flip the radio to “Mixed purpose crawlers will continue to be allowed.”

That one line changes what Cloudflare’s dashboard is now. It isn’t just a CDN with some optional security switches anymore. It’s a policy layer for AI search, sitting in front of every site that ever configured Cloudflare and forgot about it.

What “mixed purpose” means in Cloudflare’s AI Crawl Control

Cloudflare’s bot categories now split into two useful buckets.

Training-only bots — CCBot (Common Crawl), Bytespider (ByteDance), anthropic-ai (Anthropic’s training crawler — not the same as ClaudeBot), cohere-ai (Cohere). These scrape content to train future models. You get no attribution, no referral traffic, no citation. Blocking them is a defensible business decision.

Mixed-purpose bots — GPTBot (OpenAI/ChatGPT), ClaudeBot (Anthropic/Claude), PerplexityBot (Perplexity), Google-Extended (Google AI Overviews). These do both. They scrape for training and fetch pages in real time when a user asks the LLM a question and it needs to cite something. Block these and your site stops showing up when someone asks ChatGPT “which coffee grinder should I buy under $200.” You disappear from the answer even though your SEO is fine.

The radio in the AI Crawl Control panel has two options. The default that ships on new zones — and quietly rolls out on old ones — is “Mixed purpose crawlers will be blocked on September 15.” Not “already blocked.” Not “review this setting.” Just: on that date, silently, they will be. No email, no banner, no changelog entry a small-business owner will ever read.

If you want AI search visibility, you flip it to “Mixed purpose crawlers will continue to be allowed.” Two clicks.

The Cloudflare crawler log that caught the dashboard lying

The interesting part came when I pulled the 24-hour crawler log for this random little site. Not a Reddit-scale property. Just a small business page nobody’s heard of.

ClaudeBot           73 hits
OpenAI stack        47 hits
Googlebot           32 hits
PerplexityBot       19 hits
CCBot               10 hits
misc AI/other       26 hits
                   ----
                    207 total in 24h, 0 blocked

Two hundred and seven AI-related crawler requests in a day, on a site with modest human traffic. Every one served 200 OK. Zero blocks — even though the per-bot toggles in the panel were all switched to ON.

Why? Because the per-bot switches have a scoping dropdown almost nobody opens. This site’s toggles were scoped to “pages with ads,” which is the shipped default. This site had no ads. So the toggles were doing nothing. From the dashboard it looked defended. From the log it was wide open.

That’s the pattern I keep hitting with Cloudflare’s AI stack. The visual state of a switch does not tell you what’s happening. You have to look at the logs, and you have to know what to look for.

Bot Fight Mode and AI Labyrinth: the two toggles that look SEO-safe and aren’t

Two more settings on the same site were configured the way security-conscious guides recommend, and both were actively hurting AI visibility.

Bot Fight Mode. Marketed as “block bad bots.” It also trips retrieval-time AI fetchers — ChatGPT-User and Perplexity-User, the specific user-agents that show up when a user asks an LLM to actually fetch and quote your page in real time. Bot Fight Mode challenges them, they fail the challenge, your citation never happens. Turn it off if AI answers matter to you.

AI Labyrinth. Cloudflare’s honeypot for training scrapers — generates fake content to poison model training runs. It’s a fun idea for a personal blog nobody wants trained on. On a business site that wants LLM citations, it will also trap retrieval bots that follow the honeypot links, which contaminates the model’s picture of your site. Off.

The clean posture for a small business that wants to show up in AI answers is close to the opposite of the “harden everything” reflex: allow retrieval, allow mixed-purpose, tolerate training-only bots (or block only the ones that actually respect robots.txt — CCBot does, most don’t).

Here’s the tiered robots.txt I ended up shipping. It explicitly waves in the retrieval agents that drive citations and blocks the pure-training crawlers that give you nothing back:

# Retrieval agents — welcome
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /

# Training-only — no thanks
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Belt and braces. The Cloudflare radio is the actual enforcement layer; robots.txt is the polite request the well-behaved bots will honor.

llms.txt: the setting that actually doesn’t matter

Every LLM-SEO thread and every AI-search vendor deck for the last year has been pushing llms.txt — the “robots.txt for AI” convention. Drop a plaintext file at your root telling models what your site is about, and they’ll cite you more.

I looked at the data before adding one. Ahrefs sampled 137,000 sites in May 2026 — 97% of the ones with llms.txt files drove zero traffic from it. A SE Ranking study across roughly 300,000 domains found no statistically significant correlation between having the file and getting cited by AI. Zero major provider has publicly confirmed they read it. The one legit use case is docs sites feeding coding assistants — Cloudflare’s own developer docs ship one, and that’s essentially why the convention exists.

I added one anyway, because on Cloudflare Pages it’s a one-line change: drop llms.txt in /public/, deploy. Free insurance, not a ranking lever. If you’re spending real time on it, spend that time on schema and on making sure your 404s return 404s instead.

The CDN used to be a place you cached JavaScript. It’s a distribution decision layer now, and one of the levers it controls is whether large language models are allowed to see you at all.

— One night of auditing

The Cloudflare Pages 404 problem nobody talks about

Speaking of 404s — this site had none. Not zero missing URLs. Zero actual 404 responses. Cloudflare Pages’ default SPA-style routing was catching every missing path and serving the homepage with HTTP 200. Every broken image, every mistyped URL, every crawler probe returned “the homepage exists here” to Googlebot and to ClaudeBot.

For Google that’s a soft-404 signal that gets ignored eventually. For AI crawlers, which are much less mature about it, it means the model builds a picture of your site where the homepage is the answer to every question. That’s not what you want to be indexed as.

Fix on Astro-on-Pages is a src/pages/404.astro with <meta name="robots" content="noindex"> in the head. Pages auto-detects the emitted 404.html at build time and starts returning real 404s. Two minutes.

Cloudflare’s new role as an AI-search policy layer

Cloudflare sits in front of roughly a fifth of the web. The dashboard I audited belongs to someone who has never once thought about AI crawlers, never read a GEO post, and never opened the AI Crawl Control panel. On September 15, without them doing anything, their site’s ability to be cited by ChatGPT and Perplexity flips off.

That’s a new kind of infrastructure risk. It isn’t a bug. It’s a policy decision Cloudflare is making on behalf of every zone that hasn’t explicitly opted out, and it will hit a lot of small businesses whose owners never knew they were opted in.

Cloudflare AI Crawl Control FAQ

Is Cloudflare blocking ChatGPT? Not yet, and not on every site automatically. Cloudflare’s AI Crawl Control panel has a default — “Mixed purpose crawlers will be blocked on September 15” — that will block GPTBot (ChatGPT), ClaudeBot (Claude), and PerplexityBot on September 15, 2026, unless you flip it to “continue to be allowed.” Check Security → AI Crawl Control in your dashboard to see which option is selected for your zone.

What is Cloudflare AI Crawl Control? AI Crawl Control is the panel in the Cloudflare dashboard (Security → AI Crawl Control) where site owners set policy for AI bots — training-only crawlers like CCBot and mixed-purpose crawlers like GPTBot, ClaudeBot, and PerplexityBot. It’s Cloudflare’s dedicated AI-bot policy layer, separate from general bot management.

What happens on September 15, 2026? Cloudflare’s default setting for “mixed purpose” AI crawlers — GPTBot, ClaudeBot, and PerplexityBot — switches from allowed to blocked on every zone that hasn’t explicitly opted out. Sites that don’t change it stop being retrievable when ChatGPT, Claude, or Perplexity fetch a page to answer a user’s question, even though normal Google rankings are unaffected.

How do I stop Cloudflare from blocking AI crawlers? Open the Cloudflare dashboard, select your zone, go to Security → AI Crawl Control, and change the mixed-purpose-crawlers radio button to “will continue to be allowed.” Also confirm any per-bot toggles aren’t scoped to “pages with ads only” — that scope silently disables them on ad-free pages — and turn off Bot Fight Mode and AI Labyrinth if AI citations matter to you.

Does llms.txt help you get cited by AI search? Not meaningfully. Ahrefs sampled 137,000 sites in May 2026 and found 97% of the ones with an llms.txt file got zero traffic from it; an SE Ranking study of roughly 300,000 domains found no statistically significant correlation between having the file and AI citations. Spend the time on schema and real 404s instead.

Should I block CCBot and other training-only crawlers? Yes, that’s defensible. CCBot, Bytespider, anthropic-ai, and cohere-ai scrape content to train future models and give you no attribution, referral traffic, or citation back. Blocking them in robots.txt or Cloudflare’s per-bot toggles costs nothing in AI search visibility.

What to do this week

If you run anything on Cloudflare — Pages, a Workers site, a plain zone in front of Vercel — go look at the AI Crawl Control panel this week. Flip the mixed-purpose radio if you want to stay in AI answers. Verify the per-bot toggles aren’t quietly scoped to “pages with ads only.” Turn Bot Fight Mode and AI Labyrinth off if AI citation matters more to you than a small reduction in bot noise. Ship a real 404. Skip the llms.txt if you’re short on time.

The interesting SEO work in 2026 isn’t at the content layer anymore. A lot of it is one radio button most small-business owners have never seen.