AI Crawler Census

AI Crawler Census

See if ChatGPT & Perplexity can read and cite your site

A free tool to check whether ChatGPT, Perplexity, and Google's AI answers can read and cite your site. Backed by an original crawler-blocking census of the top 5,000 sites' robots.txt rules (September 2026). No signup required.

Technology · Business & Finance

Features

Checks your site's robots.txt against real AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more)

Backed by an original census of the top 5,000 sites' robots.txt rules

Shareable result page and an embeddable badge

Includes a free llms.txt validator

Use Cases

Site owners checking whether ChatGPT or Perplexity can cite their content

SEO/GEO practitioners auditing a client's AI-crawler exposure

Publishers deciding whether to allow or block AI training and search crawlers

Betula Assistant

Your second brain that gets things done. Call or text me.

Comments

reese-2

Built this after finding that 44% of sites blocking GPTBot also block OAI-SearchBot without realizing it's a different crawler with a different job. Wanted a free, no-signup way for anyone to check their own site in seconds, backed by real census data instead of a guess.

Upvote Reply

Premium Products

IZIPAY

IZIPAY is a crypto-funded virtual and physical card service

ego lite

Fastest browser for AI agents

PrompTessor

Generate, optimize, and manage AI prompts in one space

Ephedrine Powders

Ephedrine powder for sale, Ephedrine HCL, Buy Ephedrine powd

NexPept

Lab-tested research peptides for Canadian laboratories

Zinn Hub® Freelance Marketplace

The Fairer Freelance Marketplace

Velo Workspace

Disposable Linux/macOS VMs on Mac, with local AI Bridge

TaroTown

AI Tarot Readers That Remember the Whole Story

Shora

Meditation for how you feel right now

Friday

https://askfriday.io/

AirlineStream.dev

Airline logos for your app, delivered by API

CalculatorFox

Free online, embeddable calculators

AI Tools List

Discover, compare, and explore the best AI tools

RELIABLE REFRIGERANT GAS FOR HVAC & REFR

Freon Refrigerant gas cylinders

Face2social

Face search across public social media profiles

StarterKitPro

StarterKitPro is your 90% done SaaS starter kit - all in Next.js

Lockedweb - Porn Blocker

Porn blocker with no instant off switch.

Mailvex

Build on-brand emails with AI and your brand kit

BillingHub

Pay per invoice, not per month. Credits never expire.

VoiceDrop

AI-Powered Ringless Voicemail & SMS

Reply200

Handle every comment: grow organic reach, sell under ads

FindTourGuide

Find the Right Tour Guide. Plan Better Trips.

Backdrop

Live in-call sales assistant

ToolScope

Discover AI tools and create unique AI characters

Continua AI

AI-powered investigations, grounded in evidence.

Wowmade.ai

AI Video, Image & Music Generator in One Creative Platform

Kipper

Ask your finance system questions in plain English

Renoir AI

Create stunning images, videos, and audio with AI.

Buy

Betula Assistant

Your second brain that gets things done. Call or text me.

Zens AI

AI customer support that keeps improving your SaaS

EntreDash

Spark to business, your personal innovation OS

reese-2 · Follow

Makers

reese-2 · Follow