Robots.txt builder
Pick an explained preset or write your own rule groups, and your browser builds a robots.txt you can copy or download, with plain warnings and a tester that says whether a path is allowed for a given crawler. The tool never visits your site or reads your current file. It is a robots.txt generator that explains each rule before you publish it.
Loading the tool…
Processing and privacy: The result is calculated inside your browser and your data is not sent to any server.
How to use it
- Pick a preset that matches what you want, or start from the group that is already there.
- Edit the groups: one user-agent per group, one path per line under Disallow and Allow. Paths start with / and are case-sensitive.
- Add Sitemap lines as absolute URLs if you have them, and a Crawl-delay only if you need one. Google ignores it.
- Read the warnings and try a path in the tester to confirm the result before you publish.
- Copy or download the file and upload it to the root of the domain, so it answers at https://example.com/robots.txt.
Example
Applying “block specific folders” and then adding a group for a crawler called BadBot produces: User-agent: * with Disallow: /admin/, Disallow: /cart/, Disallow: /checkout/ and Disallow: /search, then a blank line, then User-agent: BadBot with Disallow: /.
In the tester, the path /admin/help/index.html for Googlebot comes back blocked by /admin/. Add Allow: /admin/help/ and it becomes allowed, because the longest matching rule wins. The pattern /*.pdf$ matches /files/a.pdf but not /files/a.pdf?x=1, because $ anchors the end of the path.
Limits and caveats
- robots.txt is a request, not enforcement. Well-behaved crawlers follow it; abusive ones simply ignore it. The file stops nobody from opening a page.
- It is not access control, and it is published for everyone at /robots.txt. Writing Disallow: /secret-admin/ advertises that path instead of hiding it. Real protection is a login or a server-side rule.
- Blocking a path does not guarantee removal from search results, and can prevent it: a crawler that may not fetch the page will never see a noindex tag on it. To remove a page, leave it allowed and add noindex, or request removal in your webmaster tools.
- The tester follows RFC 9309: the group is chosen by a case-insensitive match of the product token, the longest matching rule wins, Allow wins a tie, and * and $ are supported. Real crawlers differ in the details, and simple crawlers may not support patterns at all.
- Paths are compared as written: %XX sequences are not decoded before matching, so /café/ and /caf%C3%A9/ are different paths here. Case matters too: /Admin/ does not block /admin/.
- Comments (#) are not written, and a # inside a path or a crawler name is rejected, because everything after it would be read as a comment and break the line. The Yandex-specific Host and Clean-param directives are not written either.
- Tested limits: up to 20 groups, 20 user-agent names per group, 200 rule lines per group, 20 Sitemap lines, and Crawl-delay between 0 and 3,600 seconds. Google reads only the first 500 KiB of a robots.txt file.
- The tool does not visit your site, read your existing robots.txt, or check that the paths you type exist.
Questions about this tool
Does blocking a page in robots.txt remove it from Google?
No. Blocking stops crawling, not indexing. If other sites link to the URL it can stay in the results, usually with no description. Worse, the block stops the crawler seeing any noindex tag on the page, so the URL can stay indefinitely. The correct fix is to leave the URL allowed and add noindex, protect it with a login, or request removal in your webmaster tools.
Can I hide private pages by putting them in robots.txt?
No. The file itself is public and anyone can read it at https://example.com/robots.txt, so every path you list is announced to everyone. Use a login or a server-side rule for private content, and keep robots.txt for saving crawl effort on pages that have no value in search.
Where does the file go, and does a subdomain need its own?
Only at the root of the domain: https://example.com/robots.txt. A file in a subfolder is ignored. Rules apply per host and per protocol, so https://shop.example.com needs its own file, and http:// versus https:// (or a different port) count as separate origins.
How do I create a robots.txt file?
Pick a preset or write your rules, then download the file and upload it to the root of the domain so it answers at https://example.com/robots.txt. The simplest file, which lets every crawler fetch every page, is User-agent: * followed by an empty Disallow: line; that is what the “Allow everything” preset writes. Add a Sitemap: line with your sitemap URL if you have one, and try any path in the tester before you publish.
What does User-agent: * with Disallow: / mean?
Those two lines ask every crawler to stay away from the whole site: User-agent: * matches all crawlers, and Disallow: / blocks every path, because every path starts with a slash. That suits a staging site that is not live yet, but it is a common mistake on a real site. An empty Disallow: with no path means everything is allowed.