Robots.txt Checker
Free tool to test if a URL is blocked by robots.txt
Paste your robots.txt, pick a bot and a URL, and see whether the URL is allowed or blocked, which rule decided it, and any errors. Free, no sign-up. It cannot fetch a live file from another domain, so you paste the contents. Google handles * and $ wildcards but not every crawler does, so confirm in Search Console too.
Features
Allowed or blocked verdict, deciding rule shown, syntax errors flagged, user-agent matching with wildcard fallback, longest rule wins with Allow winning ties, free, no sign-up.
Use Cases
Testing a Disallow rule before deploying it, finding out why a page is not being crawled, checking a staging robots.txt.
Comments
Showing the deciding rule is the right call. One pitfall I'd love the tool to flag: when a bot matches its own user-agent group (say Googlebot), it ignores the * group completely, so the Disallow lines under * no longer apply to it. People add a Googlebot group for one tweak and accidentally unblock everything else. A note like "this bot uses group X, the * group is ignored" next to the verdict would catch that.
Premium Products
Sponsors
Buy
Makers
Makers
Comments
Showing the deciding rule is the right call. One pitfall I'd love the tool to flag: when a bot matches its own user-agent group (say Googlebot), it ignores the * group completely, so the Disallow lines under * no longer apply to it. People add a Googlebot group for one tweak and accidentally unblock everything else. A note like "this bot uses group X, the * group is ignored" next to the verdict would catch that.
Premium Products
New to Fazier?
Find your next favorite product or submit your own. Made by @FalakDigital.
Copyright ©2026. All Rights Reserved