Robots.txt Generator & Tester
SEO & web · Done in 3 steps
- 1Set crawler rules
- 2
- 3
About
Build a robots.txt with user-agent groups, sitemaps and one-click AI crawler blocking (GPTBot, ClaudeBot, CCBot…), then test any URL against it.
How to use it
- 1Set rules for all crawlers (*): add paths to Disallow and exceptions to Allow.
- 2Add groups for specific bots, or tick AI crawlers to block them from the whole site.
- 3Add your sitemap URL and download robots.txt.
- 4Switch to “Test a URL” to check whether a path is allowed for a given bot, using your rules or a pasted file.
- You give
- User-agent groups, paths, sitemap URLs, or an existing robots.txt
- You get
- robots.txt file and an allowed/blocked verdict with the matching rule
- Good to know
- robots.txt is advisory: reputable crawlers obey it, bad actors don’t. AI crawler names are listed as published by their operators but change over time, so verify them. Blocking crawling doesn’t remove pages already indexed; use noindex for that. The tester implements Google’s matching rules; other engines may differ slightly.
- Privacy
- Processed on your device. Nothing is uploaded, stored or logged.
- Last tested
- 2026-10-28
FAQ
Does blocking GPTBot or ClaudeBot hurt my Google rankings?
No. They are separate crawlers. Google Search uses Googlebot; Google-Extended only controls whether your content is used for Gemini. Blocking AI crawlers doesn’t change normal search indexing.
If a page matches both Allow and Disallow, which wins?
The rule with the longest matching path wins. If they’re the same length, Allow wins. The tester shows which rule decided the result.
Why is my Googlebot group ignoring the * rules?
A crawler uses only the most specific group that names it. If you have a Googlebot group, Googlebot ignores the * group entirely, so repeat any shared rules there.
Go further with AI
Content calendar workflow
Plan a month of content from goals and pillars, with formats per channel, a realistic cadence and a production schedule you can actually keep.
ChatGPT · Claude · Gemini
Customer interview synthesis
Turn interview transcripts into evidence-backed themes, jobs-to-be-done statements and opportunities, with quotes and counts rather than vibes.
Claude · ChatGPT · Gemini
Competitor comparison worksheet
Compare competitors on criteria buyers actually use, with evidence for every cell and an honest view of where you win and lose.
ChatGPT · Claude · Gemini
Landing page critique
Get a prioritised, conversion-focused critique of a landing page's copy and structure, judged from the point of view of the visitor you're paying for.
ChatGPT · Claude · Gemini