Are UK hotel websites blocking AI agents without knowing it?
UKUpdated By Akkrev
Security settings, CDN bot protection and robots.txt rules can turn away AI crawlers and guests' agents. Many UK hotels don't know theirs are doing it.
Often, yes. Hotels rarely decide to block AI. It happens through defaults: a hosting company’s bot protection, a security plugin, a robots.txt copied from somewhere else, or a booking engine vendor’s settings. Nothing looks broken to human visitors, so nobody notices.
Three layers that can block agents
robots.txt
A text file that tells crawlers what they may read. Some website templates and plugins add rules that block AI crawlers by name. Blocking OpenAI’s OAI-SearchBot, for example, can keep a hotel out of ChatGPT search answers. See robots.txt.
Bot protection at your host or CDN
Services that sit in front of websites can challenge or block anything that looks automated. Settings like “block AI bots” or strict “bot fight” modes catch scrapers, and also AI crawlers and agents acting for real guests. This layer overrides robots.txt: a crawler you allow in robots.txt can still be blocked here.
Your booking engine
Booking engine vendors protect their systems from rate scraping and card testing. Some may show CAPTCHAs or challenges on searches, which would stop agents at the first step. In our pilot of 28 UK hotels no challenge appeared before prices, but settings vary and can change. The hotel often has no idea, because the setting belongs to the vendor.
Why it matters now
Google’s John Mueller has said that one of the new basics of the agent era is “not blindly blocking agentic browsers”. As guests hand more of their booking to agents, a blocked agent is a lost direct booking, or one that goes to an OTA at commission.
Why nobody notices
When a security service blocks a scraper, nothing changes for human visitors. When it blocks a guest’s agent, the agent quietly goes elsewhere. No error reaches the hotel. The only sign is a gap: fewer AI-referred visits than expected, or AI answers that rely on third-party listings rather than the hotel’s own site.
Training versus search versus agents
It helps to think about three separate decisions:
- Training crawlers (GPTBot, ClaudeBot): whether future models learn from your site.
- Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot): whether AI answers can cite you today.
- Agents and user-triggered fetchers (ChatGPT-User, Claude-User, Google-Agent): whether a guest’s assistant can open your pages and book.
Many hotels block all three without meaning to block any of them.
What this means for AI recommendations
A site that blocks AI crawlers also limits what AI assistants can learn from it. The assistant then describes the hotel from OTA listings and reviews alone, with their prices and their links. Blocking can quietly hand your AI answers to Booking.com.
How to check
- Open
yourhotel.co.uk/robots.txtand look for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or Google-Extended next to “Disallow”. - Ask your web host or agency which bot protection is switched on and whether it treats AI agents differently.
- Ask a real AI agent to open your booking page. If it reports a challenge, a CAPTCHA or an error, that’s what guests’ agents see too.
Our free agent check covers step 3 and tells you where the block happened.
Questions
How do I know if my hotel website blocks AI agents?
Check robots.txt for AI crawler names next to 'Disallow', ask your web host or CDN what bot protection is on, and ask a real AI agent to book your hotel and see whether it gets a challenge or an error.
Does blocking GPTBot stop my hotel appearing in ChatGPT?
Not directly. GPTBot is for training. ChatGPT search uses OAI-SearchBot, and live page visits use ChatGPT-User. Blocking those is what removes a hotel from ChatGPT answers and agent visits.
Can my booking engine block AI agents even if my website doesn't?
Yes. Booking engines often run on the vendor's own domain with their own bot protection, which the hotel may not control or even know about.