AI crawlers do three different jobs: some gather training data, some build search indexes, and some fetch a single page live because a user just asked about it. A business that wants customers should welcome the second and third kinds, since they're the ones that turn into citations, links, and recommendations. Whether to allow training bots is a judgment call, and they don't pose a threat.
Why does it matter which bot is which?
The names in your logs and robots.txt lead to very different outcomes. Blocking the wrong one can drop you from AI search results without protecting you from anything, and if you allow everything without checking, you've made a decision without knowing what it was. This list takes about ten minutes to read and helps you avoid both.
OpenAI: GPTBot, OAI-SearchBot, and ChatGPT-User
GPTBot is OpenAI's training crawler. It collects public web content that may inform future models. Blocking it does not affect whether ChatGPT can search for or cite you today.
OAI-SearchBot builds and maintains the index behind ChatGPT's web search. If you want to appear, with a link, when ChatGPT answers "who should I call about a leaking roof" questions, this is the bot to welcome.
ChatGPT-User is the on-demand fetcher: when someone in a conversation asks about your site or drills into a source, this agent retrieves the page in real time. Blocking it means the assistant can't load your page while a potential customer is asking about you.
Anthropic: ClaudeBot, Claude-SearchBot, and Claude-User
ClaudeBot is Anthropic's main crawler, gathering content that may be used in training Claude models. Anthropic also operates Claude-SearchBot, which supports search indexing, and Claude-User, which fetches pages when a Claude user asks about them. It's the same three-part pattern as OpenAI's, and the same advice applies about which ones to allow.
Perplexity: PerplexityBot and Perplexity-User
Perplexity is an answer engine that cites sources more prominently than most, which makes it especially relevant to local visibility. PerplexityBot indexes for its search results; Perplexity-User fetches live when a user's question calls for your page. Infrastructure providers have publicly accused Perplexity of fetching content in ways that sidestep crawler blocks, which the company disputes. However that turns out, a local business probably still wants Perplexity's citations.
What about Google-Extended?
Google-Extended is different from the rest because it isn't a crawler at all. It's a robots.txt token that ordinary Googlebot honors, controlling whether your content can be used for Gemini training and related purposes. Two things follow. First, you'll never see "Google-Extended" fetching pages in your logs. Second (and this trips people up constantly), blocking it does not remove you from AI Overviews, which are built on Google's regular search index. Your presence there depends on ordinary Googlebot access and ordinary SEO.
Other AI crawlers and tokens
- CCBot crawls for Common Crawl, a nonprofit web archive whose datasets have fed the training of many models. Because so many models draw on those datasets, blocking it reduces your presence in future training data across the industry.
- Meta-ExternalAgent collects content for Meta's AI training; Meta also runs user-triggered fetchers for its assistants.
- Applebot-Extended, like Google-Extended, is a token rather than a bot. It governs whether content Applebot already crawled can train Apple's models.
- Bytespider, ByteDance's crawler, has a reputation for ignoring robots.txt, so robots rules do little and a firewall rule is the only reliable way to keep it out.
What allow/block policy makes sense?
For a business whose goal is customers, the rule of thumb is to allow anything that can put your name in an answer. In practice:
- Allow the search-index and user-fetch bots: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User.
- Default to allowing the training bots (GPTBot, ClaudeBot, CCBot), because a model's background knowledge also shapes its recommendations. Blocking them is a matter of principle more than marketing, and it mostly makes sense for businesses that sell content itself.
- Leave Google-Extended alone unless you've thought specifically about Gemini training; it has no bearing on your search ranking or AI Overview presence.
- Enforce with your firewall as well as robots.txt for bots that don't behave. Then check that the same firewall isn't also blocking bots you meant to allow, which is a common mistake.
Whatever policy you choose, check that it's actually in effect. robots.txt records what you intend and your server logs record what happens, and the two often disagree. A GEO audit looks for where they differ, and Speak Local automates that check for every crawler in this guide.
