# What is an AI crawler?

> An AI crawler is a bot that AI companies use to fetch web pages, for training models, building AI search indexes or fetching a page live for a user's question.

_UK and US · Updated 2026-10-27 · Akkrev, https://akkrev.com/glossary/ai-crawler/_

An **AI crawler** is an automated visitor sent by an AI company to read web pages. There are three broad kinds, and they matter differently to a hotel:

- **Training crawlers** collect text to train models. Examples: [GPTBot](https://akkrev.com/glossary/gptbot/), [ClaudeBot](https://akkrev.com/glossary/claudebot/). Blocking them doesn't remove you from AI answers today, but it affects what future models know.
- **Search crawlers** build the index AI assistants cite. Examples: [OAI-SearchBot](https://akkrev.com/glossary/oai-searchbot/), Claude-SearchBot, PerplexityBot. Blocking these can remove a hotel from AI answers.
- **User-triggered fetchers** visit a page because a person asked about it right now. Examples: ChatGPT-User, Claude-User, Perplexity-User, [Google-Agent](https://akkrev.com/glossary/google-agent/).

## Why hotels block them by accident

Hosting companies, CDNs and security plugins often treat all bots alike. A setting meant to stop rate scrapers can also turn away the search crawler that would have recommended the hotel, or the agent trying to book it. Checking [robots.txt](https://akkrev.com/glossary/robots-txt/) and [bot management](https://akkrev.com/glossary/bot-management/) settings together is the only way to know.

## Why it matters for AI visibility

AI crawlers are how assistants learn about your site. Blocking them means AI describes you from other sources instead.

---

Akkrev gets hotels, restaurants and cafés recommended by AI assistants. Get started: https://akkrev.com/start/