On September 15, 2026, Cloudflare changed how it handles AI bots. If your contractor website runs behind Cloudflare, and most small business sites do, this change could be quietly cutting you off from every AI search engine that matters. ChatGPT, Perplexity, and Claude could all be hitting a wall when they try to read your site.
The fix is not complicated. But most trades business owners do not know the problem exists, and that is the issue.
What Cloudflare changed on September 15
Cloudflare now splits AI bot traffic into three categories: Search, Agent, and Training. Starting September 15, new domains get a different default setting. Bots classified as Training or Agent are blocked automatically. Bots classified as Search are allowed through. (Source: Cloudflare Docs, September 15, 2026 update.)
That sounds reasonable until you look at what falls into each bucket. GPTBot, which ChatGPT uses to crawl the web for answers, can get caught in the Agent or Training category depending on how your Cloudflare settings are configured. Google-Extended, which Google uses for AI Overviews and AI Mode, is supposed to be in the Search category. But older Cloudflare accounts that had the blanket "Block AI Bots" toggle enabled are still blocking everything, including the bots that feed AI search results.
Cloudflare confirmed this in their announcement: the old "Block AI Bots" setting blocked all AI crawlers as one group. The new system separates them, but only for new domains and only if you update your settings. Existing domains keep their old configuration unless you manually switch. (Source: Cloudflare Blog, "Content Independence Day," July 2026; Cloudflare Docs, updated September 15, 2026.)
For a plumber whose web developer turned on "Block AI Bots" two years ago to stop scrapers, that setting is still active. It is still blocking GPTBot. And that plumber is invisible to ChatGPT.
Why 73% of businesses are already invisible
Research from Search Score found that 73% of businesses are effectively invisible in AI search. Not because their content is poor, but because AI crawlers cannot reach their websites. Cloudflare's Bot Fight Mode, default bot protection settings, and overly aggressive WAF rules are the primary culprits. (Source: Search Engine Journal, "AEO and GEO in 2026," citing Search Score research.)
A trades business can spend thousands on a new website, fill out every Google Business Profile field, collect hundreds of reviews, and still never appear in a ChatGPT recommendation. The reason is not the content. The reason is a bot protection rule that says "no" before the AI ever sees the page.
This is the most common AI visibility problem FirstCall finds when auditing trades businesses. The website exists and the Google ranking is fine, but the AI crawlers get a 403 or a CAPTCHA challenge instead of the page content. ChatGPT moves on to the next business in the list, and the contractor never knows it happened.
How to check if Cloudflare is blocking AI bots on your site
If your website uses Cloudflare, log in to the Cloudflare dashboard and check three things.
First, look at the Bot settings. Go to Security, then Bots. If "Bot Fight Mode" is enabled, it may be challenging AI crawlers with CAPTCHAs or blocking them outright. The new system replaced this with "Bot Preference Sync," but older accounts still have the original toggle. If it is on, AI bots that should be reading your site are getting blocked.
Second, check the WAF rules. Go to Security, then WAF. Look for any custom rules that block bots based on user-agent strings. If a rule blocks GPTBot, PerplexityBot, ClaudeBot, or Google-Extended, those crawlers cannot read your site. They will not appear in AI answers.
Third, test your robots.txt. Open yourdomain.com/robots.txt in a browser. If you see "Disallow: /" under GPTBot, PerplexityBot, or Google-Extended, those crawlers are told to stay away. They comply. They do not crawl pages that disallow them.
What to allow and what to block
The goal is not to let every bot through. Scrapers and bad actors exist. But the bots that feed AI search engines need access.
Allow these bots:
- GPTBot (ChatGPT)
- Google-Extended (Google AI Overviews and AI Mode)
- PerplexityBot (Perplexity)
- ClaudeBot (Claude)
- CCBot (Common Crawl, feeds several AI models)
Block what you want. But understand that blocking AI search bots means blocking AI search visibility. A contractor who blocks GPTBot is telling ChatGPT to skip their business. That is a choice, but it should be an informed one.
What to do if your site is not on Cloudflare
If your site does not use Cloudflare, the same logic applies to robots.txt. Check the file. Some website builders, hosting platforms, and WordPress security plugins add bot-blocking rules without asking. A plugin called "All in One SEO" or "Wordfence" can quietly add rules that block AI crawlers.
The test is simple. Ask ChatGPT "what services does [your business name] offer?" If ChatGPT does not know, or gives a generic answer, the crawler cannot reach the site. That is the signal.
The cost of doing nothing
A homeowner asks ChatGPT for a plumber in Bergen County. ChatGPT reads the web and finds three plumbing businesses with clear websites, strong reviews, and accessible pages, then recommends one. The other two it found are the ones whose sites are crawlable.
The plumber whose site is blocked by a Cloudflare setting never enters the conversation. The work could be excellent and the reviews could be strong, but a bot rule said "no" and the AI moved on.
Every day that rule stays active is a day the business is invisible to AI search. The September 15 Cloudflare change made this worse for anyone who had the old settings in place. But the fix takes ten minutes.
Run a free AI visibility audit
FirstCall Visibility checks whether AI crawlers can reach a trades website, what the AI engines see when they get there, and what to fix. The free Spot Check report covers ChatGPT, Google AI Mode, Perplexity, and Gemini. Run the free audit at firstcallvisibility.com.
For a full guide on which directories ChatGPT reads when recommending contractors, see which directories ChatGPT reads. For why a good Google ranking does not guarantee ChatGPT visibility, see why your Google ranking doesn't mean ChatGPT recommends you. To audit your own trades business right now, see how to audit your trades business on ChatGPT.
FAQ
Does Cloudflare block ChatGPT from reading my website?
Yes, if the "Block AI Bots" toggle or Bot Fight Mode is enabled in your Cloudflare settings. ChatGPT's crawler, GPTBot, gets blocked at the edge and never reaches your site. The September 15, 2026 update split AI bots into categories, but older accounts keep their original settings unless manually updated.
How do I check if my website is blocking AI crawlers?
Check three things: Cloudflare's Bot settings (Security, then Bots), WAF custom rules for blocked user-agents, and your robots.txt file. If any of these block GPTBot, Google-Extended, PerplexityBot, or ClaudeBot, those AI engines cannot read your site.
Does blocking AI bots hurt my Google ranking?
No. Google's main search crawler, Googlebot, is separate from Google-Extended, which feeds AI Overviews and AI Mode. Blocking Google-Extended blocks AI search features but not traditional search ranking. However, blocking AI bots means your business will not appear in AI-generated answers, which is a growing share of how homeowners find contractors.
Should I block AI training bots but allow AI search bots?
Yes. Cloudflare's new Bot Preference Sync lets you allow Search bots while blocking Training and Agent bots. This gives AI search engines access to your site for answering queries without letting training crawlers scrape your content for model development. The setting is in the Cloudflare dashboard under Bot settings.
How long does it take for AI engines to find my site after unblocking?
Most AI crawlers re-crawl within two to four weeks. ChatGPT, Perplexity, and Google AI Mode will pick up the site on their next crawl cycle. There is no submission process for AI engines the way there is for Google Search Console. You unblock the bots and wait for the next crawl.