Robots.txt Generator

robots.txt tells well-behaved crawlers which parts of your site they're allowed to request — it's a set of instructions, not a security mechanism. It does not remove pages from search results by itself, and it does not stop crawlers that choose to ignore it, which some do. For pages you genuinely need excluded from search results, a noindex meta tag or HTTP header is the correct tool; robots.txt only manages crawl behavior.

A very common mistake is disallowing a page in robots.txt while expecting that to deindex it — a page blocked from crawling can sometimes still appear in results (typically with no description) if other sites link to it, because Google discovered the URL from those links rather than from crawling the page itself.

Beyond traditional search crawlers like Googlebot and Bingbot, a growing list of AI-related crawlers (GPTBot, CCBot, and others) now request permission through the same robots.txt mechanism, each identified by its own user-agent string. Many site owners now write separate rule blocks so they can, for example, allow search engine crawling while restricting AI-training crawlers, or vice versa — a distinction that didn't exist a few years ago.

Any user-agent not explicitly matched by a rule block falls back to the wildcard User-agent: * block, so that block should reflect your general policy, with specific bots overridden individually where you want different treatment.

Blocking /wp-admin/ but forgetting the trailing slash, which can unintentionally also block a page literally named /wp-admin (rare, but it happens). Disallowing a broad path like / by accident during a staging-to-production migration and forgetting to revert it — this alone has taken entire sites out of search results before. Listing a sitemap URL that's outdated or returns a 404, which doesn't break crawling but wastes the file's one genuinely useful signal to bots.