Robots.txt Checker
Robots.txt Checker
Free tool to test if a URL is blocked by robots.txt
Paste your robots.txt, pick a bot and a URL, and see whether the URL is allowed or blocked, which rule decided it, and any errors. Free, no sign-up. It cannot fetch a live file from another domain, so you paste the contents. Google handles * and $ wildcards but not every crawler does, so confirm in Search Console too.
Features
Allowed or blocked verdict, deciding rule shown, syntax errors flagged, user-agent matching with wildcard fallback, longest rule wins with Allow winning ties, free, no sign-up.
Use Cases
Testing a Disallow rule before deploying it, finding out why a page is not being crawled, checking a staging robots.txt.
Comments
Premium Products
Find your next favorite product or submit your own. Made by @FalakDigital.
Copyright ©2026. All Rights Reserved