If your site sits behind a web application firewall, CDN, or bot-protection service, there's a real chance it's turning AI crawlers away without ever telling you. Many of these tools challenge or block unfamiliar bots by default — and AI crawlers fail those challenges — so your site can look perfect to you and to Google while being a locked door to ChatGPT, Perplexity, and Claude.
Why would a firewall block AI crawlers?
Because that's its job, as originally defined. Bot protection was built to stop scrapers, spammers, vulnerability scanners, and credential-stuffing scripts. It works by spotting non-human traffic and challenging or dropping it. AI crawlers are non-human traffic — new names, unfamiliar patterns, sometimes aggressive crawl rates — so they trip the same wires the bad bots do.
The defaults have also shifted deliberately. Cloudflare, which fronts a huge share of the web, now blocks known AI crawlers by default for new customers, framing it as protecting content from uncompensated training. Whatever you think of that argument for publishers, it's a strange default for a local business whose entire goal is to be found and recommended. Security plugins on WordPress, managed WAF rulesets, and "block automated traffic" toggles all pile onto the same effect. Nobody at your company decided to hide from AI. The defaults decided.
How do you tell if it's happening to you?
Three checks, in increasing order of certainty.
1. Run a fetch test
Use curl to request your homepage with an AI crawler's user agent — GPTBot, ClaudeBot, PerplexityBot — and note the response. A 200 with your real content is a pass. A 403, a 503, or a page asking to verify a browser is a red flag. One nuance: some firewalls verify that a claimed crawler is coming from the vendor's real IP ranges, so a spoofed-agent test from your laptop can be blocked while the genuine bot sails through. Treat a failed curl as a reason to dig, not final proof.
2. Read your server logs
Your origin access logs are the ground truth for what reached the server. Search them for the crawler names — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot — and look at two things: whether they appear at all, and what status codes they received. Crawlers showing up and collecting 403s is a confirmed block at your server.
The trickier result is silence. Zero AI bot hits can mean the bots aren't interested — or that a CDN or firewall upstream is dropping them before they ever reach your origin, in which case your logs will never show the evidence. Absence in origin logs is not absence of demand.
3. Check the CDN or firewall dashboard
If you're behind Cloudflare or a similar service, its security events log shows blocked requests by user agent. Filter for the AI crawler names and you'll see exactly which rules fired and how often. This is where upstream blocks that never touched your server become visible.
Should you allow AI crawlers or keep blocking them?
It's a genuine tradeoff, but a different one than the headlines suggest. The strongest case for blocking belongs to publishers whose content is the product. If you sell journalism or original research, letting training bots ingest it for free is a real cost, and blocking is a coherent position.
A local business is playing a different game. Your content isn't the product — it's the advertisement. Being described accurately in an AI answer is the whole point. The bots that build search indexes and fetch pages for users mid-conversation are the ones that produce citations and referrals, and blocking them is roughly equivalent to blocking Googlebot two decades ago. Even blocking training-only bots cuts both ways for you, because a model's background knowledge of local providers shapes what it recommends. Bandwidth, the other argument for blocking, is a rounding error for a typical local site.
How do you let the right bots in without dropping your guard?
You don't have to choose between security and visibility. Modern firewalls let you allowlist specific verified crawlers — Cloudflare, for instance, maintains a verified-bots category you can permit while everything unverified still gets challenged. The sane setup for a business that wants customers: explicitly allow the major AI search and assistant crawlers, keep aggressive protection on everything else, and express any finer-grained policy in robots.txt, since legitimate bots honor it.
Then verify: rerun the curl test, watch the logs for a week, and confirm the bots are fetching pages and getting 200s. Crawl access is the unglamorous foundation of GEO — nothing else you do matters if the engines can't get in the door. It also drifts: a CDN update or a new security plugin can quietly reinstate a block months later, which is why continuous crawler-access monitoring is part of what platforms like Speak Local watch alongside the visibility those crawls eventually earn. Your firewall should guard the store, not turn away the customers' scouts.
