UK hotels blocking ChatGPT-User in robots.txt
UKUpdated By Akkrev
One pilot hotel blocked about 20 AI crawlers, including ChatGPT-User, yet its booking engine worked for our agent. What blocking does and doesn't do.
A robots.txt file tells automated visitors which parts of a website they may read. Many hotel websites now include rules for AI companies’ bots. Some block them all.
In our pilot of 28 independent UK hotels, one hotel’s robots.txt blocked about 20 AI crawlers by name. The list included GPTBot, ClaudeBot, PerplexityBot, CCBot and ChatGPT-User. We checked the live file on the day of the test.
The surprising part
That hotel’s booking engine worked perfectly. Our scripted agent found the booking link, opened the engine and reached prices for the requested dates. It passed.
The reason is simple. The booking engine was on its provider’s domain, which has its own robots.txt. And robots.txt is a request, not a lock. A browser-based agent acting for a user may not read it at all.
So does blocking matter?
Yes, in other ways. Different bots do different jobs:
- Training crawlers such as GPTBot and CCBot collect pages that may be used to train models.
- Search crawlers such as OAI-SearchBot help assistants find and cite pages in answers.
- User-triggered fetchers such as ChatGPT-User fetch a page because a person asked about it.
Blocking the second and third types can mean an assistant can’t read your room descriptions, policies or prices when a guest asks about you. It may rely on online travel agencies or reviews instead, or recommend a competitor whose site it can read. That’s a visibility problem, before any booking starts. OpenAI explains the different bots on its crawler overview page.
Why hotels end up with these blocks
Few hoteliers write robots.txt by hand. Blocks often arrive through:
- A web agency adding a standard list.
- A security or SEO plugin with an “block AI bots” setting.
- A hosting or CDN default, sometimes switched on site-wide.
- A copy of another site’s file.
The hotel may have made a deliberate choice about training. Blocking search and user-triggered fetchers as well is often a side effect.
A choice worth making on purpose
Some hotels don’t want their content used to train models. That’s a fair position. It’s possible to treat training bots, search bots and user-triggered fetchers differently. The important thing is knowing which ones you block and what each block costs. Read more in are UK hotel websites blocking AI agents?.
What this means for AI recommendations
Blocking ChatGPT’s crawlers means ChatGPT learns less from your own site and more from others. You can still be recommended, but on other sites’ terms and links.
How to check
Open yoursite.co.uk/robots.txt in a browser. Look for lines naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot or Google-Extended, followed by “Disallow: /”. Then ask your web agency who added them and why. Our free agent check can look at this too.
Questions
What is ChatGPT-User?
It's the user agent OpenAI uses when ChatGPT fetches a page because a user asked it to, rather than to crawl the web for training or search. OpenAI documents its crawlers and user agents on its bots page.
If a hotel blocks AI crawlers, can an agent still book it?
In our pilot, yes: the booking engine sat on another domain with its own rules, and our scripted browser completed the search. But blocking can still stop assistants from reading the hotel's pages, which affects whether it's found and recommended.
Who adds these blocks?
Often a web agency, a security plugin or a hosting provider, sometimes without the hotel knowing. Some block lists are copied between sites.
Does robots.txt apply to booking engines?
Each domain has its own robots.txt. A booking engine on the provider's domain follows the provider's rules, not the hotel's.