Why does AI crawler indexability matter?

If your site blocks GPTBot, PerplexityBot, or ClaudeBot in robots.txt, those engines cannot fetch your pages and cannot cite you in AI answers. Google-Extended controls whether Google can use your content to train Gemini and generate AI Overviews. Blocking these bots is a common, invisible cause of zero AI visibility.

What's the difference between robots.txt and a live block?

robots.txt is a polite request — well-behaved bots respect it. A live block (403, 401, 429, or a Cloudflare / WAF challenge) rejects the bot at the network layer regardless of robots.txt. This tool checks both: the robots rule for each user agent and the live HTTP response when we identify as that bot.

Should I allow all AI crawlers?

For most brands that want AI visibility — yes. The exception is training-only crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) if you have a strategic reason to withhold your content from LLM training while still being cited live. Live-fetch bots (ChatGPT-User, Perplexity-User, Claude-Web, OAI-SearchBot) are the ones that produce citations, so those should almost always be allowed.

The tool shows 'blocked' but I never added a rule. Why?

Common causes: (1) a Cloudflare 'Block AI Scrapers' rule set at the edge, (2) a WAF challenge that returns 403 to unknown user agents, (3) a Vercel / Netlify header rule, or (4) a security plugin on WordPress. Check your CDN or firewall before editing robots.txt.

Free tool

Is Your Site Blocking ChatGPT, Perplexity & Claude?

Free AI crawler access test for any website. We fetch your robots.txt and request your homepage as thirteen major AI bots — GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, Claude-Web, Google-Extended, Applebot-Extended, Bytespider, CCBot, Meta-ExternalAgent, Amazonbot, and cohere-ai — then show you exactly which engines are allowed to read, index, and cite your pages in ChatGPT, Perplexity, Claude, and Google AI Overviews.

Domain

We check the root path (/) against each bot's user-agent. Root-path blocking is the most common cause of missing AI citations.

If you're invisible in AI, you're losing clients right now.

See exactly how your company appears across AI, search, and investor research — and uncover the hidden gaps costing you trust and deals.