ToolSahla, Tools Sahla

robots.txt: what it blocks and what it doesn't

Your store's cart, checkout and internal search pages are not worth crawling. A robots.txt file asks search engines to skip them. But many people think it hides or protects pages, and it does not.

Published · 2 min read

A request, not a lock

Well-behaved crawlers like those of the major search engines follow the file; abusive ones ignore it. The file is published for everyone at /robots.txt, so a line blocking /secret-admin/ advertises that path instead of hiding it. Real protection is a login or a server-side rule.

Blocking does not remove a page from results

If you block crawling of a page, a search engine never sees a noindex tag on it, so it may stay in results. To remove a page from results, leave it allowed and add noindex, or request removal in your webmaster tools.

Worked example

A group for all crawlers that blocks /admin/, then allows /admin/help/.

The path /admin/help/index.html becomes allowed, because the longest matching rule wins.

And the pattern /*.pdf$ matches /files/a.pdf, but not /files/a.pdf?x=1, because $ marks the end of the path.

User-agent: * Disallow: /admin/ Allow: /admin/help/

How to do it

  1. Open the Robots.txt builder and pick an explained preset, such as blocking specific folders, or write your own groups.
  2. Add your sitemap link.
  3. Try any path in the tester to see whether it is allowed for a given crawler, and which rule decided it.
  4. Copy or download the file and upload it to the root of your site.

Common mistakes

  • Paths are case-sensitive: /Admin/ does not block /admin/.
  • A lone Disallow: / under User-agent: * blocks crawling of your whole site. Check it carefully before uploading.
  • Google reads only the first 500 KiB of the file.

Add the link to the sitemap you build with the XML sitemap generator. For the pages you do want found, check their title and description in results with the Search snippet preview.

Common questions

Does robots.txt protect my store's admin page?

No. The file is a request that only well-behaved crawlers follow, and it is public. Protection comes from a password or a server-side rule.

Does the tool read my current file?

No. It does not visit your site or read your current file; it builds a new one from what you enter.

How do I remove a page from search results?

Leave it allowed for crawling and add a noindex tag, or request removal in your webmaster tools.