AI Crawler Census
See if ChatGPT & Perplexity can read and cite your site
A free tool to check whether ChatGPT, Perplexity, and Google's AI answers can read and cite your site. Backed by an original crawler-blocking census of the top 5,000 sites' robots.txt rules (September 2026). No signup required.
Features
Checks your site's robots.txt against real AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more)
Backed by an original census of the top 5,000 sites' robots.txt rules
Shareable result page and an embeddable badge
Includes a free llms.txt validator
Use Cases
Site owners checking whether ChatGPT or Perplexity can cite their content
SEO/GEO practitioners auditing a client's AI-crawler exposure
Publishers deciding whether to allow or block AI training and search crawlers
Comments
Useful tool and the 5,000-site robots.txt census backing it is a nice touch. One thing a lot of people get wrong: blocking Google-Extended does not remove you from AI Overviews in Google search, it only affects training/grounding, while GPTBot and PerplexityBot rules directly decide whether you get cited inside those tools. If your report spelled out that difference per crawler, people would know exactly what they gain or lose before editing their robots.txt. I'd try this on a couple of sites before trusting any crawler-blocking advice.
Premium Products
Sponsors
Buy
Comments
Useful tool and the 5,000-site robots.txt census backing it is a nice touch. One thing a lot of people get wrong: blocking Google-Extended does not remove you from AI Overviews in Google search, it only affects training/grounding, while GPTBot and PerplexityBot rules directly decide whether you get cited inside those tools. If your report spelled out that difference per crawler, people would know exactly what they gain or lose before editing their robots.txt. I'd try this on a couple of sites before trusting any crawler-blocking advice.
Premium Products
New to Fazier?
Find your next favorite product or submit your own. Made by @FalakDigital.
Copyright ©2026. All Rights Reserved