Robots.txt Generator
Build robots.txt rule blocks one bot at a time, and get a correctly formatted file out the other end — no need to remember the exact directive syntax or worry about a stray character breaking the file.
robots.txt
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://example.com/sitemap.xmlUpload this file to your site's root — it must be reachable at yoursite.com/robots.txt exactly.
What Is Robots.txt Generator?
This tool assembles a robots.txt file from a set of rule blocks you define visually — a User-agent, its Disallow and Allow paths, an optional crawl-delay, and a shared Sitemap declaration — instead of asking you to hand-write the plain text format from memory.
It's aimed at the moment you're setting up crawl rules for a new site, or restructuring an existing file to treat a specific bot differently, and want to avoid the small formatting mistakes that are easy to introduce by editing the raw file directly.
How to Use This Tool
- 1Pick a user-agent for your first rule block (use * for a rule that applies to every crawler that doesn't have its own more specific block).
- 2List the paths that agent should be blocked from, one per line, and any exceptions in the Allow field.
- 3Add another rule block for any bot you want to treat differently — for example, restricting an AI crawler while leaving search engines untouched.
- 4Add your sitemap URL, then copy the generated file or download it directly.
Understanding Your Results
Multiple rule blocks
Each block you add becomes its own User-agent group in the output. Bots that don't match any specific group you've defined fall back to the wildcard (*) block, so make sure that block reflects your general policy rather than being left overly permissive or overly restrictive by accident.
The generated file
The live preview updates as you edit, so you can see exactly what will be uploaded before you commit to it. Download it directly as robots.txt, or copy the text if you're pasting it into a CMS or hosting panel's file editor.
Common Problems & How to Fix Them
Forgetting the file has to sit at the domain root
A correctly generated robots.txt only works if it's placed at exactly yoursite.com/robots.txt — not in a subfolder, and not on a different subdomain than the one you're trying to control. Double-check the upload location after downloading.
Blocking more than intended with a broad wildcard rule
A rule like Disallow: / in the wildcard group blocks the entire site from every crawler that doesn't have a more specific group — a common accidental cause of a site vanishing from search results after a robots.txt update. Review the full generated file before uploading, not just the block you were actively editing.
SEO Best Practices
Start from your actual crawl goals rather than a generic template — list the specific folders or bots you need to treat differently, and keep everything else open by default.
Test the result with a robots.txt tester against a handful of real URLs before treating the file as final, especially if you're managing rules for more than one bot.
Revisit the file whenever you add a new section to the site or start using a new crawler-facing service, since robots.txt tends to go stale quietly if nobody owns keeping it current.
Frequently Asked Questions
Do I need a robots.txt file if I have nothing to block?
No — if you don't have one, crawlers generally assume everything is crawlable. A robots.txt file becomes useful once you have admin areas, staging paths, or specific bots you want to manage differently from the rest.
Can I generate different rules for Googlebot and other bots in one file?
Yes — add a separate rule block for each user-agent you want to treat differently. Any crawler not matched by a specific block falls back to the wildcard (*) block.
Does adding a Sitemap line help my SEO directly?
It's a minor convenience signal that helps crawlers discover your sitemap faster, not a ranking factor on its own. It's still good practice to include it.