Skip to content
Akkrev
Get started

Methodology

Updated

Pilot round: 20 October 2026

The first round used a simpler set-up than the full method below, and we report it separately.

  • Sample: 28 independent UK hotels, tested on 20 October 2026, most with one or two attempts.
  • Agent: a scripted browser that reads pages as text, like a text-based agent. Not ChatGPT agent, Gemini in Chrome or another named product; their results could differ.
  • Task: from the hotel's homepage, find the booking engine and reach a room list with prices for one night, two adults, on a fixed Friday and a fixed Monday. Where the engine supports it, dates were passed in the web address. No guest details, card details or bookings were ever entered.
  • Scores, 0 to 2 each (total 12): discoverable from the homepage, reaches the booking engine, dates honoured, rates readable as text, friction (cookie banner only scores 2; a passive CAPTCHA badge or optional sign-in scores 1; a hard block or error scores 0), and speed.
  • Outcome: pass means readable rates for the chosen dates; partial means the engine was reached but rates weren't, or the dates were unclear; fail means no working engine was reached.
  • Reported by engine only where at least two hotels used it. Hotels are not named.

Limits: small samples, UK only, one agent type, and rate accuracy not checked. Load times include tool overhead, so compare them between rows rather than reading them as absolute speeds.

The task

Each agent gets the same plain request, for example: "Book a double room for two adults for two nights from [date] at [hotel], on the hotel's own website. Stop before paying and tell me the total price and cancellation policy." Dates are 2 to 6 weeks ahead, on nights the hotel shows as available.

Agents

At least two general-purpose agents per hotel, chosen from those consumers can use at the time of testing, such as ChatGPT agent, Gemini in Chrome, Claude and Perplexity Comet. We record the product and date of each run.

Hotels

3 to 5 independent hotels, B&Bs, inns or vacation rentals per booking engine, split across the UK and US where the engine is used in both. We identify the engine from the hotel's live booking page.

What we record

  • Completion: pass (reached a priced room and the payment page), partial (found availability or price but couldn't continue), fail (couldn't get a price).
  • Speed: minutes and steps from the hotel's home page to a priced room.
  • Understanding: whether the total price, taxes, fees and cancellation policy the agent reported match the booking page.
  • Blockers: bot walls, CAPTCHAs, widgets the agent couldn't operate, logins and pop-ups.

Scoring

Each engine's score is the average across its hotels and agents. Results are published as dated snapshots and re-run each quarter.

Limits

  • Agents change often, and the same agent can behave differently on two runs. A result is a snapshot, not a verdict.
  • A hotel's own settings, themes and add-ons can change the result on the same engine.
  • We stop at the payment page and never make real bookings, so payment steps aren't scored.

Vendor responses

Booking engine vendors can email [email protected]. We publish responses next to results and re-test after fixes.