Skip to content
Answer Rankings Answer Rankings Answer Rankings

CCBot: Common Crawl and the Training Data Question

CCBot is the crawler of Common Crawl, a non-profit that publishes a free snapshot of the web every month. That dataset has been training fuel for a large share of the world’s language models. Allowing CCBot is a bet on being present in future models; blocking it has almost no effect on today’s citations.

User agent

Section titled “User agent”
CCBot/2.0 (https://commoncrawl.org/faq/)

Robots.txt token: CCBot. It honours robots.txt and crawls politely.

What makes CCBot different

Section titled “What makes CCBot different”

The other crawlers in the directory each serve one engine you can name. CCBot serves an open dataset consumed by research labs, startups and established AI companies alike. Consequences:

  • Allowing it seeds your content into the default corpus many new models start from — the widest possible, slowest-acting distribution.
  • Blocking it doesn’t remove you from anything currently deployed, and the majors now crawl for themselves (GPTBot, ClaudeBot), so the cost of blocking is lower than it was in 2023.

Allow or block?

Section titled “Allow or block?”
# Maximum future-model presence
User-agent: CCBot
Allow: /


# Opt out of open training datasets
User-agent: CCBot
Disallow: /

This is the one genuinely discretionary call in the directory. Visibility-first businesses should allow; content licensors reasonably block. Either way, make the choice deliberately — plenty of sites block CCBot only because it arrived bundled in a copy-pasted “block AI” robots.txt snippet.

Section titled “Related”