Skip to content
Akkrev
Get started

Free AI crawler check

Enter a website to see which AI crawlers and agents its robots.txt lets in, whether the homepage answers or shows a bot challenge, and whether it has an llms.txt file. Results in a few seconds, nothing to install.

What this checks

  • robots.txt rules for 15 crawlers and agents from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Meta and Common Crawl, for your homepage.
  • The homepage, fetched like a normal browser from our server: the status code, and whether it shows a bot challenge.
  • Robots meta tags and headers that tell search engines not to index or use the page.
  • llms.txt, a plain-text guide some sites publish for AI assistants.

What it doesn't check

Firewalls and bot protection can treat real AI crawlers differently from our check, because they also look at where a request comes from. A clean result here doesn't prove an agent can book. That needs a real agent to try, which is what our free agent check does. To track whether AI assistants recommend you at all, see our AI visibility plans.

The crawlers

  • GPTBot (OpenAI, training): Collects pages to train OpenAI models.
  • OAI-SearchBot (OpenAI, search): Finds pages to show and link in ChatGPT search.
  • ChatGPT-User (OpenAI, user-triggered): Visits a page when a ChatGPT user or agent asks it to.
  • ClaudeBot (Anthropic, training): Collects pages to train Anthropic's Claude models.
  • Claude-SearchBot (Anthropic, search): Finds pages to show in Claude's search results.
  • Claude-User (Anthropic, user-triggered): Visits a page when a Claude user asks it to.
  • PerplexityBot (Perplexity, search): Finds pages to show and cite in Perplexity answers.
  • Perplexity-User (Perplexity, user-triggered): Visits a page when a Perplexity user asks it to.
  • Google-Extended (Google, training): Controls whether Google may use pages for Gemini training and grounding. It is a robots.txt setting, not a separate crawler.
  • Googlebot (Google, search engine): Google Search, including AI Overviews and AI Mode.
  • Bingbot (Microsoft, search engine): Bing search, which also feeds Microsoft Copilot.
  • Applebot-Extended (Apple, training): Controls whether Apple may use pages to train its AI models.
  • Amazonbot (Amazon, search): Amazon's crawler, used for Alexa and other Amazon services.
  • meta-externalagent (Meta, training): Collects pages for Meta's AI models and products.
  • CCBot (Common Crawl, training): Builds the open Common Crawl dataset many AI models are trained on.

Agents and developers can call this check directly: GET /api/crawler-check?url=example-hotel.com. See the OpenAPI description.