What is ClaudeBot?
Updated By Akkrev
ClaudeBot is Anthropic's web crawler. Anthropic also runs Claude-SearchBot for search and Claude-User for pages fetched live when a user asks.
ClaudeBot is the crawler Anthropic uses to collect public web content for training Claude models. Like other major AI crawlers, it respects robots.txt.
Anthropic runs separate agents for separate jobs:
- ClaudeBot: training data.
- Claude-SearchBot: improving search results in Claude.
- Claude-User: fetching a page live because a user asked Claude about it.
Why it matters to hotels
When a guest asks Claude to compare hotels or check a hotel’s policies, Claude-User fetches the page right then. If the hotel’s site blocks it, Claude has to rely on other sources, which may be out of date.
As with GPTBot, a hotel can make different choices for training and search. The point is to choose deliberately, not to block everything by accident through a security setting. See bot management.
Why it matters for AI visibility
ClaudeBot is one of the crawlers that feeds what Claude knows about the web. Allowing it helps Claude describe you from your own site.