AI Crawlers and Cloudflare: Should Your Business Block or Allow AI Bots?
Cloudflare now lets every plan allow or block AI bots by behavior: search, agent and training. Here is what each control does and a decision guide for businesses that want leads from AI answers.
Should your business block AI crawlers in Cloudflare? For most small and mid-sized businesses that want customers, the short answer is: allow AI search crawlers, decide deliberately about training crawlers, and check what your Cloudflare zone is already doing, because the defaults changed in 2026. Cloudflare now groups AI bots by behavior (Search, Agent and Training) and lets every plan, including Free, allow or block each group. Blocking everything protects your content but can also keep your company out of the AI answers that prospects increasingly read. This guide explains the controls in plain language and ends with a decision guide.
If you want the basics of Cloudflare's firewall first, read our guide to protecting your website with Cloudflare WAF.
Three kinds of AI bots, three different trade-offs
Cloudflare classifies AI bots by what they do on your site, not by a single "AI" label. Its documentation describes three behaviors available to all customers:
| Behavior | What it does | What blocking it means for you |
|---|---|---|
| Search | Collects or indexes content so an AI assistant can answer questions later | Your pages may stop appearing or being cited in that assistant's answers |
| Agent | Acts in real time for a person, such as a chat assistant fetching a page or a browser-use agent | An assistant cannot open your page when a user asks it to |
| Training | Crawls content to train or fine-tune a model | Your content is not used for training by bots that honor the block |
The difference matters because each AI company runs separate crawlers. OpenAI, for example, documents OAI-SearchBot for showing sites in ChatGPT search, GPTBot for crawling content that may be used to train its models, and ChatGPT-User for visits a person triggers inside ChatGPT. OpenAI states that these settings are independent: blocking GPTBot opts you out of training without removing you from ChatGPT search. Google uses a robots.txt token, Google-Extended, to control use of crawled content for Gemini model training and grounding; Google states it does not affect inclusion in Google Search.
Some crawlers are mixed-purpose, used for both search and training. Cloudflare's changelog notes that multi-purpose crawlers combining Search and Training are affected by a Training block, so blocking Training can reduce visibility in that provider's AI search as well.
Cloudflare's bot controls, explained
Bot Fight Mode (Free plan)
Bot Fight Mode is Cloudflare's free tool that challenges traffic matching known bot patterns. It is simple to turn on (Security Settings, filter by Bot traffic), but Cloudflare lists important limits: it protects the whole domain, it cannot be bypassed with WAF custom rules or Page Rules, and it may challenge API or mobile app traffic. If your site has webhooks, a mobile app or integrations that call it, test carefully or skip it.
Super Bot Fight Mode (Pro and above)
Super Bot Fight Mode is included with Pro, Business and Enterprise plans. It sorts traffic into groups (definitely automated, likely automated and verified bots) and lets you choose an action for each. Unlike Bot Fight Mode, it supports skip rules, so you can exempt your own API clients or monitoring tools.
Verified bots
A verified bot is one Cloudflare has confirmed is transparent about who it is and what it does. Cloudflare requires honest self-identification (a cryptographic Web Bot Auth signature, a published IP list or reverse DNS) and non-abusive behavior, including obeying robots.txt. Since July 1, 2026, signed agents are part of the same group. Allowing verified bots is how search engines such as Googlebot and Bingbot keep reaching your site while unknown automation is challenged.
The AI-specific controls
AI bot policies (replacing "Block AI bots")
On July 1, 2026, Cloudflare announced that the single Block AI bots toggle is being replaced by AI bot policies with one setting per behavior. In the dashboard: Security Settings, filter by Bot traffic, then Configure AI bot policies. For Search, Agent and Training you can choose:
- Block (on all pages)
- Block on pages with ads (Cloudflare detects ad units automatically)
- Allow (do not block)
Cloudflare's documentation says the legacy Block AI bots option is deprecating on September 15, 2026, and that from that date new domains get updated defaults: Training and Agent bots are blocked on pages that display ads, while Search stays allowed. Coverage of the change disagrees on whether existing zones were also changed, so do not assume: open the page and check what your zone shows today.
For a typical business site without ads, "Block on pages with ads" has little practical effect. If you want a real decision, choose Block or Allow explicitly.
AI Crawl Control
AI Crawl Control (formerly AI Audit) is available on all plans. It shows which AI crawlers visit your site and how often, lets you allow or block individual crawlers, and tracks which crawlers follow your robots.txt. It is the right place to start: look at real traffic before you block anything. It also includes Pay Per Crawl, which lets site owners charge AI crawlers for access; Cloudflare's documentation lists it as a private beta, so most small businesses can ignore it for now.
Managed robots.txt and content signals
robots.txt is a request, not a lock: compliance is voluntary. Cloudflare's managed robots.txt (available on all plans) adds AI crawler directives for you, based on your AI bot policies, when Bot Preference Sync is enabled. If you already have a robots.txt, Cloudflare places its managed rules above yours and serves one file. It also includes the Content Signals Policy, which defines three uses (search, ai-input and ai-train) so you can state preferences in a machine-readable way. Pair it with AI bot policies or AI Crawl Control if you want blocking enforced at the edge rather than requested.
A decision guide for businesses that want leads from AI answers
If people find you by asking an assistant "Who does IT support in Miami?" or "Which CPA firms in Coral Gables handle small businesses?", you want to be readable and citable. A practical default for a B2B service company:
- Search: Allow. This keeps your pages eligible to appear in AI search answers. Keep regular search engines allowed through verified bots.
- Agent: Allow on marketing pages. When a prospect asks an assistant to check your services page or contact details, you want that fetch to succeed. Protect logins, portals and forms with rate limiting or WAF rules instead.
- Training: your call. Blocking training crawlers protects your writing from being used to build models, but may also affect mixed-purpose crawlers' search features. If your pages are marketing content meant to be read, allowing it is a reasonable choice. If your content is the product (research, courses, paid guides), blocking it makes more sense.
- Look before you block. Review AI Crawl Control for a week. If a specific crawler is hammering your server, block that crawler, not the whole category.
- Keep robots.txt and Cloudflare consistent. A robots.txt that welcomes a crawler while Cloudflare blocks it (or the opposite) makes troubleshooting harder.
- Protect what is private. Client portals, invoices and admin areas should require a login and be excluded from indexing; bot settings are not access control.
- Write content worth citing. Clear service pages, a real service area, plain answers to common questions and accurate contact details matter more than any crawler setting. Our custom websites work is built around this.
Re-check these settings when Cloudflare announces changes; its bot products are renamed and reorganized often.
If you would like help reviewing your Cloudflare bot and AI crawler settings, or building a site that AI assistants can understand and cite, request a free IT assessment. Core Florida IT is an independent IT provider serving businesses across Miami-Dade and South Florida, and our AI business apps team works with these tools every day.
Sources
- Bots: AI bot behaviors (Cloudflare Docs)
- Block AI bots and AI bot policies (Cloudflare Docs)
- New options to manage AI traffic, July 1, 2026 (Cloudflare Changelog)
- AI Crawl Control (Cloudflare Docs)
- Managed robots.txt (Cloudflare Docs)
- Bot Fight Mode (Cloudflare Docs)
- Super Bot Fight Mode (Cloudflare Docs)
- Verified bots (Cloudflare Docs)
- Overview of OpenAI crawlers (OpenAI)
- Google's common crawlers and Google-Extended (Google Search Central)
