If your site sits behind a web application firewall, CDN, or bot-protection service, there's a real chance it's turning AI crawlers away without telling you. Many of these tools challenge or block unfamiliar bots by default, and AI crawlers fail those challenges. Your site can look perfect to you and to Google while staying locked to ChatGPT, Perplexity, and Claude.
Why would a firewall block AI crawlers?
Because that's what it was designed for. Bot protection was built to stop scrapers, spammers, vulnerability scanners, and credential-stuffing scripts. It spots non-human traffic and challenges or drops it. AI crawlers are non-human traffic with new names, unfamiliar patterns and sometimes aggressive crawl rates, so they trip the same wires the bad bots do.
Some defaults have also changed on purpose. Cloudflare, which fronts a huge share of the web, now blocks known AI crawlers by default for new customers, framing it as protecting content from uncompensated training. Whatever you think of that argument for publishers, it's a strange default for a local business that wants to be found and recommended. Security plugins on WordPress, managed WAF rulesets, and "block automated traffic" toggles all add to the same effect. Nobody at your company decided to hide from AI. The defaults decided for you.
How do you tell if it's happening to you?
There are three checks, from least to most certain.
1. Run a fetch test
Use curl to request your homepage with an AI crawler's user agent (GPTBot, ClaudeBot, PerplexityBot) and note the response. A 200 with your real content is a pass. A 403, a 503, or a page asking to verify a browser is a red flag. One caveat: some firewalls verify that a claimed crawler is coming from the vendor's real IP ranges, so a spoofed-agent test from your laptop can be blocked while the genuine bot gets through. Treat a failed curl as a reason to dig further. It isn't final proof.
2. Read your server logs
Your origin access logs are the ground truth for what reached the server. Search them for the crawler names (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot) and look at two things: whether they appear at all, and what status codes they received. If crawlers show up and get 403s, your server is blocking them.
Silence is harder to read. Zero AI bot hits can mean the bots aren't interested. It can also mean a CDN or firewall upstream is dropping them before they reach your origin, and then your logs will never show the evidence. An empty origin log doesn't prove the bots never came.
3. Check the CDN or firewall dashboard
If you're behind Cloudflare or a similar service, its security events log shows blocked requests by user agent. Filter for the AI crawler names and you'll see exactly which rules fired and how often. This is where upstream blocks that never touched your server become visible.
Should you allow AI crawlers or keep blocking them?
There is a tradeoff, but it differs from the one in the headlines. The strongest case for blocking belongs to publishers whose content is the product. If you sell journalism or original research, letting training bots ingest it for free is a real cost, and blocking is a coherent position.
A local business is in a different position. For you, the content is the advertisement. Being described accurately in an AI answer is the point. The bots that build search indexes and fetch pages for users mid-conversation are the ones that produce citations and referrals, and blocking them is roughly like blocking Googlebot two decades ago. Even blocking training-only bots cuts both ways for you, because a model's background knowledge of local providers shapes what it recommends. Bandwidth, the other argument for blocking, is a rounding error for a typical local site.
How do you let the right bots in without dropping your guard?
You can keep both security and visibility. Modern firewalls let you allowlist specific verified crawlers. Cloudflare, for instance, maintains a verified-bots category you can permit while everything unverified still gets challenged. A sensible setup for a business that wants customers: explicitly allow the major AI search and assistant crawlers, keep aggressive protection on everything else, and put any finer-grained policy in robots.txt, since legitimate bots honor it.
Then verify: rerun the curl test, watch the logs for a week, and confirm the bots are fetching pages and getting 200s. Crawl access is the unglamorous foundation of GEO. Nothing else you do matters if the engines can't get in the door. Access also drifts. A CDN update or a new security plugin can quietly reinstate a block months later, which is why platforms like Speak Local keep monitoring crawler access alongside the visibility those crawls eventually earn. Keep the firewall, and make sure it lets these crawlers through.
