Robots.txt Generator

Useful that the toggle exists, and one thing worth spelling out next to it: blocking AI bots is not one decision. Training crawlers (GPTBot, CCBot, Google-Extended) and search or retrieval crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User) do different jobs, and blocking the second group means your pages cannot be cited in AI answers at all. A split option, block training but allow search bots, would cover what most product sites want. Also worth reminding people that a blanket "Disallow: /" under "User-agent: *" blocks those search bots too.

Nice and simple, and no sign-up is a plus! qq can I block AI training bots (GPTBot, CCBot) but still allow AI search bots (OAI-SearchBot, PerplexityBot)? That's the setup I would want for a product site.

The AI bot blocking toggle is what sets this apart — most robots.txt generators just handle allow/disallow paths and ignore the AI crawler question entirely. Including the sitemap URL line in the generated file is a genuinely useful touch, since that is the step people forget right after editing. One question: does the preview let you edit or reorder rules before copying the file, or do you start over if you spot a mistake?

Useful, and the AI bot toggle is what I'd use it for. From running two content sites: robots.txt only stops the polite crawlers (GPTBot, ClaudeBot, CCBot); most of the scraping we saw came from datacenter IPs with browser user-agents that never read the file. It might help to say that next to the toggle, and maybe note that Google-Extended is only a token: blocking it keeps content out of Gemini training but does not touch Googlebot or search.

No sign-up and copy-ready output is exactly right for a file you touch maybe twice a year. The AI bot blocking toggle is the reason I would pick this over writing the file by hand. One thing I would check: the AI crawler list changes fast (ClaudeBot, PerplexityBot, Bytespider all appeared recently) - how often do you update the list of agents the toggle covers?

AI bot blocking on top of the usual robots.txt rules is a nice touch — most generators stop at Googlebot/Bingbot and ignore the GPTBot/CCBot crowd. Do you auto-update the AI bot list as new crawlers show up, or is that a manual edit?

This is a handy tool for website owners who want to manage crawling and improve their technical SEO workflow. I also found some useful free SEO and website analysis tools here: https://tools.corenexis.com — worth checking out if you're looking for practical web utilities without signing up.

The optional AI bot blocking toggle is the practical bit here, since most generators skip that question. Having the sitemap line in the generated file saves a step people forget.