AI Crawlers List 2026: Every Bot That Decides Your AI Visibility

Eight crawlers decide most of your AI visibility in 2026. If you want ChatGPT, Perplexity, Claude, Gemini, Copilot or Apple Intelligence to cite your site, these are the user agents your robots.txt must not block.

Crawler	Operator	Powers	Blocking it means
GPTBot	OpenAI	ChatGPT model training + browsing	Absent from ChatGPT’s knowledge
OAI-SearchBot	OpenAI	ChatGPT Search	No citations in ChatGPT search answers
PerplexityBot	Perplexity	Perplexity answers	Invisible to Perplexity
ClaudeBot	Anthropic	Claude training + retrieval	Absent from Claude’s knowledge
Google-Extended	Google	Gemini training + grounding	Reduced Gemini visibility (Google Search unaffected)
Bingbot	Microsoft	Bing Search and Copilot	Gone from Bing and Copilot both
CCBot	Common Crawl	Training data for many models	Slow fade from future models
Applebot-Extended	Apple	Apple Intelligence training	Absent from Apple AI features

The one-minute audit

Open https://yoursite.com/robots.txt and look for any of the names above under a Disallow: / rule — or run our free AI visibility checker, which tests the six highest-impact bots in one pass and scores the rest of your AI-readiness while it’s there.

Which should you allow?

For any business that wants to be found: allow all eight. The “block AI scrapers” wave of 2023 made sense for publishers selling licensed content; for merchants, agencies and B2B companies it quietly removes you from the channels buyers now use to choose suppliers. Blocking Bingbot is the most expensive accident on the list — it takes out classic Bing Search and Copilot in one line.

There are two legitimately different calls:

• CCBot — feeds open training datasets rather than a single engine. Allow if you want maximum future-model presence; block if dataset reuse bothers you. Direct citation impact today: minimal.
• Google-Extended — controls Gemini’s use of your content, not Google Search. Blocking it does not protect your rankings; it only mutes you in Gemini.

Copy-paste: allow the citation engines

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Bingbot
Allow: /

Full syntax, edge cases and CDN gotchas: robots.txt for AI crawlers.