Which AI crawlers to allow, and what each one actually does
Not all AI bots do the same job. Some fetch pages to answer a question right now, others collect training data. Blocking the wrong one removes you from answers you wanted to appear in.
Key takeaways
- AI crawlers split into answer crawlers, training crawlers and user-triggered fetchers, with different consequences for blocking.
- Blocking answer crawlers removes you from live citations; blocking training crawlers does not.
- Check your live robots.txt after deploys, and check CDN bot protection, which blocks at the edge invisibly.
- A blocked crawler cannot read a noindex directive, so blocking is not the same as excluding.
AI crawlers fall into three functional groups, and the distinction decides what blocking one costs you. Search and answer crawlers fetch pages so an assistant can cite them in a live response. Training crawlers collect text for model training. User-triggered fetchers retrieve a specific page because someone asked about it. Blocking the first group removes you from citations. Blocking the second is a content-licensing decision. Blocking the third breaks a request a real person made.
The three groups
- Answer and search crawlers, such as OAI-SearchBot, PerplexityBot and Google-Extended in its answer capacity. These build the index an assistant draws on when citing sources. If citation is the goal, these must be allowed.
- Training crawlers, such as GPTBot, ClaudeBot, CCBot and Applebot-Extended. Allowing them contributes your content to future model training. This is a genuine business decision with reasonable answers in both directions, and it does not affect whether you are cited today.
- User-triggered fetchers, such as ChatGPT-User and Perplexity-User. These fire when a person pastes your link or asks about your page. Blocking them produces a visible failure for someone who was actively trying to read you.
A default that suits most businesses
For a company that wants to be found and cited, the sensible default is to allow all three groups on public marketing content, and to disallow everything on authenticated areas, admin routes and anything behind a login. Publishers whose product is the text itself have a different calculation and often allow answer crawlers while disallowing training ones, which is a coherent position.
What is rarely coherent is blocking everything by reflex. We regularly find sites that added a blanket disallow during a security review, then spent a quarter wondering why no assistant mentioned them.
How to check what you are actually doing
Read your live robots.txt rather than your intended one, because deployments overwrite it. Confirm each user agent you care about is named explicitly, since a rule for a wildcard user agent may be overridden by a more specific one, and precedence rules surprise people. Then verify from the outside: request a page with the relevant user agent string and check you get a 200 with real content.
Beyond robots.txt, watch for a firewall or CDN bot-protection rule blocking these agents at the edge. This is a common and invisible cause, because robots.txt looks correct while the request never reaches the origin. Check your CDN's bot management settings explicitly.
The noindex trap
One subtlety catches teams out. If you block a crawler in robots.txt, it cannot fetch the page and therefore cannot see a noindex directive in the HTML. If you want a page excluded from an AI index, allow the crawl and serve the exclusion, rather than blocking the crawl and assuming exclusion follows.
Blocking every AI bot by reflex is a decision to be absent from a surface your customers are already using.
Sources