Direct answer: allow or block?
For most SMEs who want visibility in AI answers (ChatGPT, Perplexity, Google AI Overviews): allow. Blocking mainly makes sense if you primarily live off direct visitors or ad revenue on your own content and want compensation for reuse by AI companies — think news publishers, not a local service-based SME.
The choice isn't all-or-nothing. You can make a conscious decision per bot: one crawler delivers direct visibility value with source attribution, another just trains a model with no clear value flowing back.
The main AI crawlers at a glance
Six crawlers show up most often in your server logs. Each belongs to a different AI platform, with its own purpose and default behaviour.
| Crawler | From | Purpose | Default behaviour |
|---|---|---|---|
| GPTBot | OpenAI | Trains and enriches ChatGPT answers | Crawls unless blocked via robots.txt |
| ClaudeBot | Anthropic | Training + real-time retrieval for Claude | Crawls unless blocked via robots.txt |
| PerplexityBot | Perplexity | Live answers with source citations | Crawls unless blocked via robots.txt |
| GoogleOther / Google-Extended | Training data for Gemini, separate from Google Search | Crawls unless blocked via robots.txt | |
| Amazonbot | Amazon | Alexa and AI services | Crawls unless blocked via robots.txt |
| Bytespider | ByteDance (TikTok) | Training data for TikTok and Doubao models | Often crawls aggressively; reports that robots.txt isn't always respected |
How to block an AI crawler via robots.txt
Blocking is done per user-agent in your robots.txt. The example below excludes the major AI training crawlers while Googlebot (the classic search index crawler) keeps full access:
User-agent: GPTBot — Disallow: / · User-agent: ClaudeBot — Disallow: / · User-agent: PerplexityBot — Disallow: / · User-agent: Google-Extended — Disallow: / · User-agent: Amazonbot — Disallow: / · User-agent: Bytespider — Disallow: /
Note: robots.txt is a request, not a technical lockout. Most major AI companies respect it properly, but there have been repeated reports that Bytespider doesn't always comply. For a hard block, you need an extra layer via your CDN or firewall (for example Cloudflare's AI bot blocking feature).
The trade-offs: what you gain and lose
Blocking protects your content from uncompensated training and saves bandwidth against aggressive crawlers. But it has a direct downside: blocking means being invisible in AI answers. No citations, no brand mentions in ChatGPT or Perplexity answers, no chance to be cited as a source when a potential customer asks an AI tool for advice.
For a service-based SME that's a real revenue trade-off: every AI conversation you're absent from is a potential lead going to a competitor who is accessible to these crawlers.
Recommendation by business type
SMEs seeking leads and visibility (the majority of our clients): allow all major AI bots. The visibility gain almost always outweighs the risk of content reuse.
Content publishers and news media who primarily live off subscriptions or ads on their own content: consider selective blocking, or negotiate a licence — similar to the deals major publishers and platforms like Reddit have struck with AI companies.
E-commerce with price-sensitive data: consider blocking specific price-scraping bots while keeping the major AI crawlers open for product discoverability in AI shopping answers.
Practical example: deliberately allowing
Allowing usually requires no action: without a Disallow rule for a user-agent, the bot simply crawls. For clarity, many businesses still document that explicitly in robots.txt, so future maintainers can see at a glance that the choice was deliberate, not accidentally forgotten.
User-agent: GPTBot — Allow: / · User-agent: ClaudeBot — Allow: / · User-agent: PerplexityBot — Allow: /
Note: Google-Extended is not Googlebot
Google-Extended is a separate token, specifically for training Gemini and other AI products — distinct from the classic Googlebot that indexes your site for Google Search. You can block AI training at Google without touching your organic ranking, as long as you don't block Googlebot itself. Always confirm this distinction before adding a blanket rule.
Not sure which crawlers to allow?
Request a GEO strategy call →