What is robots.txt?
Updated By Akkrev
robots.txt is a file at a site's root that tells search and AI crawlers which pages they may fetch. It can block AI assistants from a hotel site.
robots.txt is a plain-text file at a website’s root (for example yourhotel.com/robots.txt) that tells automated crawlers what they’re allowed to fetch. Each block names a crawler, such as Googlebot, GPTBot or ClaudeBot, and lists allowed and disallowed paths.
Why it matters for AI
AI companies run separate crawlers for separate jobs. OpenAI, for example, uses GPTBot for training and OAI-SearchBot for ChatGPT search results. A hotel that blocks the search crawler can disappear from ChatGPT answers, even if it never meant to.
Common surprises
- Website platforms or plugins that add AI blocks by default.
- A rule copied from a news site that blocks every AI crawler.
- A block on the booking engine’s domain, set by the vendor rather than the hotel.
Why it matters for AI visibility
robots.txt decides which AI crawlers can read your site. It’s one of the first things to check for AI visibility.
How to check
Open yourdomain/robots.txt in a browser and look for the names of AI crawlers next to “Disallow”. Remember that robots.txt is only half the story: a security service can still block crawlers that robots.txt allows. See bot management.