AI crawlers do three different jobs: some gather training data, some build search indexes, and some fetch a single page live because a user just asked about it. A business that wants customers should welcome the second and third kinds, since they're the ones that turn into citations, links, and recommendations. Training bots are a judgment call rather than a threat.
Why does it matter which bot is which?
Because the names in your logs and robots.txt lead to very different outcomes. Block the wrong one and you disappear from AI search results while "protecting" yourself from nothing. Allow everything blindly and you've made a choice without knowing it. Ten minutes with this list avoids both.
OpenAI: GPTBot, OAI-SearchBot, and ChatGPT-User
GPTBot is OpenAI's training crawler. It collects public web content that may inform future models. Blocking it does not affect whether ChatGPT can search for or cite you today.
OAI-SearchBot builds and maintains the index behind ChatGPT's web search. If you want to appear, with a link, when ChatGPT answers "who should I call about a leaking roof" questions, this is the bot to welcome.
ChatGPT-User is the on-demand fetcher: when someone in a conversation asks about your site or drills into a source, this agent retrieves the page in real time. Blocking it means a potential customer's assistant hits a wall mid-conversation with them.
Anthropic: ClaudeBot, Claude-SearchBot, and Claude-User
ClaudeBot is Anthropic's main crawler, gathering content that may be used in training Claude models. Anthropic also operates Claude-SearchBot, which supports search indexing, and Claude-User, which fetches pages when a Claude user asks about them. It's the same three-part pattern as OpenAI, with the same implications for which ones a business that wants customers should allow.
Perplexity: PerplexityBot and Perplexity-User
Perplexity is an answer engine that cites sources more prominently than most, which makes it especially relevant to local visibility. PerplexityBot indexes for its search results; Perplexity-User fetches live when a user's question calls for your page. Infrastructure providers have publicly accused Perplexity of fetching content in ways that sidestep crawler blocks, which the company disputes. However that turns out, the practical point for a local business doesn't change: you probably want the citations.
What about Google-Extended?
Google-Extended is the odd one out: it isn't a crawler at all. It's a robots.txt token that ordinary Googlebot honors, controlling whether your content can be used for Gemini training and related purposes. Two things follow. First, you'll never see "Google-Extended" fetching pages in your logs. Second (and this trips people up constantly), blocking it does not remove you from AI Overviews, which are built on Google's regular search index. Your presence there depends on ordinary Googlebot access and ordinary SEO.
Other AI crawlers and tokens
- CCBot crawls for Common Crawl, a nonprofit web archive whose datasets have fed the training of many models. Blocking it thins your presence in future training corpora across the industry, beyond any one company's.
- Meta-ExternalAgent collects content for Meta's AI training; Meta also runs user-triggered fetchers for its assistants.
- Applebot-Extended, like Google-Extended, is a token rather than a bot. It governs whether content Applebot already crawled can train Apple's models.
- Bytespider, ByteDance's crawler, has a reputation for ignoring robots.txt, which makes robots rules moot and firewall rules the only real lever against it.
What allow/block policy makes sense?
For a business whose goal is customers, the rule of thumb is: allow anything that can put your name in an answer. In practice:
- Allow the search-index and user-fetch bots: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User.
- Default to allowing the training bots (GPTBot, ClaudeBot, CCBot), because a model's background knowledge also shapes its recommendations. Blocking them is a matter of principle more than marketing, and it mostly makes sense for businesses that sell content itself.
- Leave Google-Extended alone unless you've thought specifically about Gemini training; it has no bearing on your search ranking or AI Overview presence.
- Enforce with your firewall as well as robots.txt for bots that don't behave. Then make sure that same firewall isn't silently blocking the bots you meant to welcome, which happens more often than anyone intends.
Whatever policy you choose, check that it's in effect. robots.txt says what you intend, server logs say what's happening, and the two disagree more often than you'd think. A GEO audit exists to close that gap between intended policy and observed reality, and checking it across every crawler in this guide is part of what Speak Local automates.
