Skip to content
Answer Rankings Answer Rankings Answer Rankings

ClaudeBot: Anthropic's Crawler and Your robots.txt

ClaudeBot is Anthropic’s web crawler, gathering content that improves the Claude family of models. Claude is the default assistant inside a fast-growing set of business tools — when it recommends suppliers or explains a product category, that knowledge traces back to what ClaudeBot could crawl.

User agent

Section titled “User agent”
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Robots.txt token: ClaudeBot.

Anthropic’s crawler family

Section titled “Anthropic’s crawler family”
BotJob
ClaudeBotMain crawler — training and improvement
Claude-UserLive fetches when a Claude user asks about a URL
Claude-SearchBotSearch/retrieval indexing

A ClaudeBot rule does not automatically cover the others; if you’re curating access, name each token explicitly. (Anthropic also historically used anthropic-ai and Claude-Web tokens — harmless to include for completeness.)

Allow or block?

Section titled “Allow or block?”
# Recommended
User-agent: ClaudeBot
Allow: /


User-agent: Claude-SearchBot
Allow: /

The same trade-off as GPTBot: blocking protects content from training at the cost of absence from the assistant’s knowledge. Claude’s enterprise footprint — support desks, coding tools, procurement research — makes that absence increasingly expensive for B2B businesses in particular.

Section titled “Related”