My blog is on a subdomain like blog.mystore.com. Why do the rules in the main domain's robots.txt not apply to it?

Because crawlers look for robots.txt at the root of each host separately. When they crawl blog.mystore.com they read blog.mystore.com/robots.txt and never look at mystore.com's file. Build the blog its own file with its own rules and Sitemap line, and upload it to the blog's root.

Published · 3 min read

Every host has its own file

The rule the tool states in its 'One file per host and protocol' note: the rules apply only to the host, protocol and port the file came from. In practice, these are all separate addresses, each with its own file:

  • mystore.com/robots.txt: for the main store.
  • blog.mystore.com/robots.txt: for the blog.
  • http:// versus https:// on the same host: separate origins too.

A file placed in a subfolder, such as mystore.com/blog/robots.txt, is ignored completely; the file only works from the host's root. That is a key difference: a blog in a /blog/ folder on the main domain is governed by the store's file, while a blog on a subdomain has its own.

Why the tester does not tell the domains apart

In 'Test a path' in the Robots.txt builder we entered the full URL https://blog.example.com/posts/1?x=1. The tool took the path /posts/1?x=1 and noted that only the path was tested, because robots.txt rules never look at the domain name. That is right within one file, since rules are paths only. But it also means picking the right file is up to you: test a blog link against the store's rules and the result describes a file the crawler will never read for that link.

Steps

  1. Read the blog's current file if there is one: open blog.mystore.com/robots.txt. If you get an error page, there is no file, and everything is allowed to crawlers.
  2. Build a file for the blog alone in the tool: for example a * group blocking /wp-admin/ or the blog's internal search pages, depending on what it has.
  3. Add a Sitemap line with the full address of the blog's sitemap on the subdomain, not the store's.
  4. Test real article paths to confirm they are allowed.
  5. Download the file and upload it to the blog's root, then open blog.mystore.com/robots.txt to confirm.
  6. When you want to edit the store's file, rebuild it in the tool separately so the rules never get mixed.

An example of a simple blog file that allows everything and points to its sitemap:

User-agent: * Disallow: Sitemap: https://blog.example.com/sitemap.xml

The empty Disallow line means everything is allowed; the tool writes it that way when a group has no rules, with a warning explaining so. Build the blog's sitemap from its article links in the XML sitemap generator if the blog software does not make one. If the blog is part of a store on a hosted platform, check what the platform publishes first, since you may not control the file.

Common questions

Is it a problem if the blog has no file?

No file effectively means everything is allowed. You only need one if there is something to block or a sitemap to point to.

Should the blog's sitemap go in the store's file?

A Sitemap line can point to a full URL, but it is clearer in the blog's own file. Check the search engine's documentation.

Does the tool check the blog's current file?

No, it visits no domain. Copy its rules in to test them.