# What is robots.txt?

> robots.txt is a file at a site's root that tells search and AI crawlers which pages they may fetch. It can block AI assistants from a hotel site.

_UK and US · Updated 2026-10-27 · Akkrev, https://akkrev.com/glossary/robots-txt/_

**robots.txt** is a plain-text file at a website's root (for example `yourhotel.com/robots.txt`) that tells automated crawlers what they're allowed to fetch. Each block names a crawler, such as Googlebot, [GPTBot](https://akkrev.com/glossary/gptbot/) or [ClaudeBot](https://akkrev.com/glossary/claudebot/), and lists allowed and disallowed paths.

## Why it matters for AI

AI companies run separate crawlers for separate jobs. OpenAI, for example, uses GPTBot for training and [OAI-SearchBot](https://akkrev.com/glossary/oai-searchbot/) for ChatGPT search results. A hotel that blocks the search crawler can disappear from ChatGPT answers, even if it never meant to.

## Common surprises

- Website platforms or plugins that add AI blocks by default.
- A rule copied from a news site that blocks every AI crawler.
- A block on the booking engine's domain, set by the vendor rather than the hotel.

## Why it matters for AI visibility

robots.txt decides which AI crawlers can read your site. It's one of the first things to check for AI visibility.

## How to check

Open `yourdomain/robots.txt` in a browser and look for the names of AI crawlers next to "Disallow". Remember that robots.txt is only half the story: a security service can still block crawlers that robots.txt allows. See [bot management](https://akkrev.com/glossary/bot-management/).

---

Akkrev gets hotels, restaurants and cafés recommended by AI assistants. Get started: https://akkrev.com/start/