CrawlCheck

CrawlCheck started as an internal tool: we kept getting asked "can ChatGPT see my site?" and the honest answer required manually tracing 60+ crawler user-agents through robots.txt. So we built the auditor we wanted — paste your robots.txt, get a per-bot verdict instantly, and generate a clean policy with one click. Everything runs client-side, so nothing leaves your browser. We would love feedback on the policy generator: which AI crawlers do you block today, and did the tool miss any?

Tried this on my own site - the 60+ crawler breakdown is the part I didn't know I needed, since most robots.txt audits only cover the big three bots. One suggestion: a diff view between two crawls would make re-audits after changes much faster to eyeball. Nice, focused tool.

Really useful for small sites. I just launched a fan guide and had no idea which AI crawlers my robots.txt was blocking. Do you plan to flag crawlers that ignore robots.txt too?

Congrats on the launch! Keeping everything client-side is a nice touch for something people paste config into. Does the policy generator distinguish AI training crawlers from AI search crawlers by default?

For robots.txt audits, I'd find it useful to see the exact user-agent group and winning Allow/Disallow rule behind each result. Does CrawlCheck show that trace, including wildcard fallback? It would make generated policies easier to review before publishing.

The per-bot verdict for GPTBot, ClaudeBot and PerplexityBot is handy, especially since it runs fully in the browser so I can paste a private staging robots.txt. For the policy generator, does it let you block AI training crawlers while still allowing the search ones in a single export?

Using Load example, then testing /admin/, the auditor correctly reports that the path is blocked. The warnings also describe this as lost referral traffic and say the Googlebot block removes the site from search completely. Since /admin/ is intentionally private in this example, could those messages distinguish a path-level block from a site-wide block? An “expected private path” label would help avoid suggesting that admin pages should be opened to crawlers.

Useful that it covers less obvious bots like Bytespider, not just GPTBot. When a robots.txt has both a wildcard group and a bot-specific group, does the per-bot verdict show which group actually won? That would make it easier to debug conflicting rules.

Genuinely useful - I'm working through AI-crawler policy on a few sites right now and per-bot verdicts beat manually tracing user-agents. The client-side approach is a nice trust signal too. One question: does the policy generator handle allowing search crawlers while blocking training crawlers, or is it all-or-nothing per bot?

Running everything client-side is a great choice for a robots.txt auditor, since people can test staging configs without uploading anything. Could you add a mode that fetches a live robots.txt by URL and flags bots that ignore robots.txt entirely, so users know when they need server-level blocking instead?

Could the report clearly distinguish permission in robots.txt from actual crawler access? A short note about CDN or WAF blocks and page-level noindex directives would help users understand why an allowed bot might still be unable to fetch or index a page.

Useful. The split between training crawlers and search/answer crawlers is the part most people get wrong, blocking GPTBot but forgetting OAI-SearchBot or PerplexityBot and then wondering why they never get cited. Does the verdict flag that difference per bot, or is it allow/block only?

Since the crawler list will change over time, showing its last-updated date alongside exported policies would help teams review them later. Can the per-bot audit results be exported as JSON so a site can compare its policy before and after a deployment?

Since the per-bot verdict depends on matching user-agent tokens, how do you keep the list current when vendors add new crawlers like OpenAI's separate OAI-SearchBot vs GPTBot? It would help to see a 'last updated' date next to each bot so I know whether a verdict is still trustworthy. Also curious whether the policy generator distinguishes training crawlers from ones that power live search citations, since blocking the latter can cost visibility.

Ran our own robots.txt through it. We stack 13 User-agent lines over one Allow group, and it read every one correctly, which plenty of parsers get wrong. I also stacked GPTBot, CCBot and ClaudeBot over a single Disallow, and all three came back blocked as they should. One thought: blocking only training crawlers is a deliberate choice for a lot of sites, yet it still drops the score from 100 to 95. Scoring search and assistant visibility on its own, and showing training opt-outs as a neutral setting, would stop people feeling nudged to undo it.

Ran our robots.txt through it (a wildcard Allow plus a Sitemap line): 100, all 61 crawlers allowed. Grouping the results into search, assistant, agents and training is the clearest breakdown I've seen. One suggestion: the list itself notes that Bytespider and a few others are known to ignore robots.txt, yet the summary still says CLEAN. A small count of "allowed, but known to ignore robots.txt anyway" next to the score would stop people reading 100 as "protected".

Doing the robots.txt parsing fully client-side is a nice touch for privacy. One edge case worth checking: crawlers like Google-Extended and Applebot-Extended are opt-out tokens rather than real user-agents that fetch pages, so does the per-bot verdict distinguish those from bots that actually crawl? A short note on that in the results would save people a lot of confusion.

Covering 60+ bots instead of just GPTBot is useful. Does it handle wildcard rules and Allow overriding Disallow by longest match the way Google does?

The training-vs-search split is the part most people miss: blocking GPTBot outright can also cost you the search citations that actually send traffic. A per-bot verdict is the right way to surface that. And client-side only means I can run it on any site without sending its config anywhere. Saved.

The per-bot verdict is genuinely useful — a lot of people block GPTBot and don’t realise OAI-SearchBot and ChatGPT-User are separate agents, so they lose (or keep) ChatGPT search visibility without meaning to. Suggestion: group the results into “training”, “search/answers” and “user-triggered fetch”, with one line on what blocking each group costs. Checking llms.txt next to robots.txt would also be a nice addition, since its format is easy to get wrong.

Ran this against a docs site I maintain — the client-side approach meant I could audit a staging URL behind basic auth without pasting credentials anywhere, which is exactly the right call. One feature request: an exportable GitHub Action that fails CI when a new AI crawler appears that isn't covered by the policy. The crawler landscape changes monthly; a scheduled audit would be more valuable than a one-off check.

Running the robots.txt parsing fully in the browser and distinguishing AI training crawlers from AI search crawlers is a genuinely useful split. One suggestion: let users fetch robots.txt directly by domain and also flag llms.txt or meta robots noai tags, since those often contradict what robots.txt says.

Ran our studio site's robots.txt through it: 100, 61 allowed, and the explicit Allow groups for OAI-SearchBot, Claude-SearchBot and Perplexity-User all read correctly. Two small things I noticed. First, the "Unrecognized user-agent" note lumps Slurp, Baiduspider and YandexBot (classic search engines) together with AhrefsBot, SemrushBot and MJ12bot (SEO crawlers). Splitting those into "known non-AI bot" vs "truly unknown" would stop people wondering if they have a typo. Second, the Crawl-delay note fires on "Crawl-delay: 0", which does nothing for any bot. Saying "a value of 0 has no effect, safe to remove" would be more useful there than the Googlebot reminder.