AI crawlers do three different jobs: some gather training data, some build search indexes, and some fetch a single page live because a user just asked about it. The sane policy for a business that wants customers is to welcome the second and third kinds — they're the ones that turn into citations, links, and recommendations — and to treat training bots as a judgment call rather than a threat.
Why should you care who's who?
Because the names in your logs and robots.txt map to very different outcomes. Block the wrong one and you disappear from AI search results while "protecting" yourself from nothing. Allow everything blindly and you've made a choice without knowing it. Ten minutes with the cast list beats both.
OpenAI: GPTBot, OAI-SearchBot, and ChatGPT-User
GPTBot is OpenAI's training crawler. It collects public web content that may inform future models. Blocking it does not affect whether ChatGPT can search for or cite you today.
OAI-SearchBot builds and maintains the index behind ChatGPT's web search. If you want to appear, with a link, when ChatGPT answers "who should I call about a leaking roof" questions, this is the bot to welcome.
ChatGPT-User is the on-demand fetcher: when someone in a conversation asks about your site or drills into a source, this agent retrieves the page in real time. Blocking it means a potential customer's assistant hits a wall mid-conversation with them.
Anthropic: ClaudeBot and friends
ClaudeBot is Anthropic's main crawler, gathering content that may be used in training Claude models. Anthropic also operates Claude-SearchBot, which supports search indexing, and Claude-User, which fetches pages when a Claude user asks about them — the same three-part pattern as OpenAI, with the same implications for which ones a customer-hungry business should allow.
Perplexity: PerplexityBot and Perplexity-User
Perplexity is an answer engine that cites sources unusually prominently, which makes it disproportionately interesting for local visibility. PerplexityBot indexes for its search results; Perplexity-User fetches live when a user's question calls for your page. Worth knowing: Perplexity has been publicly accused by infrastructure providers of fetching content in ways that sidestep crawler blocks, which the company disputes. Wherever that lands, the practical takeaway for a local business is unchanged — you probably want the citations anyway.
What about Google-Extended?
Google-Extended is the odd one out: it isn't a crawler at all. It's a robots.txt token that ordinary Googlebot honors, controlling whether your content can be used for Gemini training and related purposes. Two things follow. First, you'll never see "Google-Extended" fetching pages in your logs. Second — and this trips people up constantly — blocking it does not remove you from AI Overviews, which are built on Google's regular search index. Your presence there rides on ordinary Googlebot access and ordinary SEO.
Who else is in the cast?
- CCBot crawls for Common Crawl, a nonprofit web archive whose datasets have fed the training of many models. Blocking it thins your presence in future training corpora across the industry, not just one company's.
- Meta-ExternalAgent collects content for Meta's AI training; Meta also runs user-triggered fetchers for its assistants.
- Applebot-Extended, like Google-Extended, is a token rather than a bot — it governs whether content Applebot already crawled can train Apple's models.
- Bytespider, ByteDance's crawler, has a reputation for ignoring robots.txt, which makes robots rules moot and firewall rules the only real lever against it.
What's a sane allow/block policy?
For a business whose goal is customers, the rule of thumb is simple: allow anything that can put your name in an answer. Concretely:
- Allow the search-index and user-fetch bots: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User.
- Default to allowing the training bots — GPTBot, ClaudeBot, CCBot — because a model's background knowledge also shapes its recommendations. Blocking them is a philosophical stance, not a marketing strategy, and it mostly makes sense for businesses that sell content itself.
- Leave Google-Extended alone unless you've thought specifically about Gemini training; it has no bearing on your search ranking or AI Overview presence.
- Enforce with your firewall, not just robots.txt, for bots that don't behave — and make sure that same firewall isn't silently blocking the bots you meant to welcome, which happens more often than anyone intends.
Whatever policy you choose, verify it's actually in effect. robots.txt says what you intend; server logs say what's happening; and the two disagree more often than you'd think. That gap — intended policy versus observed reality — is exactly what a GEO audit exists to close, and checking it across every crawler in this guide is part of what Speak Local automates. Know who's at the door before you decide who gets in.
