Robots.txt Generator
Build a valid robots.txt with allow/disallow rules and sitemap.
Default policy
Disallowed paths
robots.txt
User-agent: * Disallow: /admin/ Disallow: /private/
Save this as robots.txt in the root of your website.
About this tool
The Robots.txt Generator builds a valid robots.txt file without you having to remember the syntax. Choose whether to allow or block all crawlers, add specific disallowed paths, set an optional crawl delay and reference your XML sitemap.
robots.txt lives at the root of your domain (example.com/robots.txt) and is the first file well-behaved crawlers request. Use it to keep admin areas, cart pages and duplicate content out of search engines — but remember it's a directive, not security.
Frequently asked questions
It prevents crawling, not indexing. A blocked URL can still appear in results if other sites link to it. Use a noindex meta tag or authentication for true removal.
At the root of your domain so it's reachable at yoursite.com/robots.txt. Subdirectory locations are ignored by crawlers.
It asks crawlers to wait the given number of seconds between requests. Bing and Yandex honor it; Google ignores it and uses Search Console settings instead.