Skip to content
Answer Rankings Answer Rankings Answer Rankings

robots.txt for AI Crawlers: The Complete 2026 Setup

Your robots.txt now has two audiences: search crawlers and AI crawlers. Most AI visibility failures trace back to this one file — usually a blanket rule added years ago. Here’s the correct 2026 setup, the syntax rules that trip people up, and the traps that block bots even when robots.txt looks fine.

The visibility-first template

Section titled “The visibility-first template”
# Search engines
User-agent: *
Allow: /


# AI answer engines — allow to be cited
User-agent: GPTBot
Allow: /


User-agent: OAI-SearchBot
Allow: /


User-agent: PerplexityBot
Allow: /


User-agent: ClaudeBot
Allow: /


User-agent: Google-Extended
Allow: /


User-agent: Applebot-Extended
Allow: /


User-agent: CCBot
Allow: /


Sitemap: https://yoursite.com/sitemap.xml

What each bot powers — and the case for each — is in the AI crawler directory.

Syntax rules that actually bite

Section titled “Syntax rules that actually bite”
  1. Groups end at the next User-agent line. Rules apply only to the group directly above them; a stray blank line doesn’t end a group, but a new User-agent does.
  2. Most specific group wins, and it wins alone. If GPTBot has its own group, it ignores the * group entirely. A common failure: User-agent: * allows everything, then a forgotten User-agent: GPTBot / Disallow: / from 2023 sits at the bottom of the file overriding it.
  3. Disallow: with an empty value means allow everything — it is not a block.
  4. Case doesn’t matter for tokens, but spelling does: GPT-Bot matches nothing.
  5. One file per host. www and apex, HTTP and HTTPS, and each subdomain serve their own robots.txt. Your store at shop.example.com is not covered by the apex file.

Selective blocking done right

Section titled “Selective blocking done right”

Want AI engines to see marketing content but not, say, your support archive?

User-agent: GPTBot
Allow: /
Disallow: /support/archive/

Path rules work for AI tokens exactly as for Googlebot: longest matching path wins.

The traps robots.txt can’t show you

Section titled “The traps robots.txt can’t show you”

Passing robots.txt doesn’t guarantee access — three layers can still block silently:

  • CDN bot management. Cloudflare and similar can challenge or block AI user agents regardless of robots.txt. If your bot-fight mode is on, GPTBot may be receiving a CAPTCHA page. Check your firewall events for the bot names.
  • WAF rules and rate limits tuned during a scraping incident and never revisited.
  • Geo-blocking — several AI crawlers exit from US IP ranges; a US block is an AI block.

Our free checker reads your robots.txt the way the bots do and flags blocked crawlers — run it after every change. For the crawlers’ own behaviour (which respect robots.txt, which fetch on user request), see the per-bot pages: GPTBot, PerplexityBot, ClaudeBot, Bingbot.

FAQ

Section titled “FAQ”

Do AI crawlers actually respect robots.txt? The major indexing crawlers do — GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, CCBot all honour it. User-triggered fetchers (ChatGPT-User, Perplexity-User) may fetch a specific URL a user asked about, like a browser would.

Should I list bots I want with Allow: /, or just not mention them? Unmentioned bots fall through to your * group. Explicit allows cost nothing and protect you from a future blanket rule accidentally catching them — we recommend explicit.