I wrote Disallow: /shop to block one folder. Why is /shop-by-brand blocked too?
Because robots.txt rules work as prefixes: /shop matches every path that starts with those five characters, including /shop-by-brand and /shopping-guide. To target only the folder, write /shop/ with a trailing slash. But note: /shop/ no longer matches the bare /shop page itself.
How a rule matches a path
A crawler does not read a rule as a folder name but as the start of a string. If the path begins with the rule character for character, it matches. So we tried several pairs in the Robots.txt builder:
- Rule /shop with path /shop-by-brand/oud: matches. The path starts with /shop.
- Rule /shop with path /shopping-guide: matches, for the same reason.
- Rule /shop/ with path /shop-by-brand/oud: no match. After /shop comes a hyphen, not a slash.
- Rule /shop/ with path /shop/oud: matches, as intended.
- Rule /shop/ with path /shop: no match. The path is shorter than the rule.
The tool warns about this itself in the 'Block specific folders' preset: /search without a slash also matches /search-results.
Three goals, three rules
- You want the folder and everything in it only: write Disallow: /shop/.
- You also want the /shop page itself, without the other paths that share its letters: add a second line, Disallow: /shop$. A $ at the end of a rule means 'the path ends here', so it matches /shop alone and not /shop/oud.
- You really do want everything starting with /shop: keep the rule without a slash, but on purpose.
With /shop/ and /shop$ together, we tested /shop-by-brand/oud again and it came back allowed, with no matching rule. This is the file:
User-agent: *
Disallow: /shop/
Disallow: /shop$
Check every nearby page before publishing
One rule can touch pages you never thought of. Before uploading the file, type every path on your site that starts with the same letters into 'Test a path', such as /shop-by-brand, /shopping-guide, /shop/oud and /shop. That way you know the rule's full effect before a crawler does.
The $ pattern is defined in RFC 9309, and the tool's own warning notes Google and Bing support it, but simpler crawlers may treat it as an ordinary character. That is why the tool shows a warning when you use it. If you are creating new paths for your site, picking names that do not share a beginning, with the URL slug generator for instance, keeps the rules simpler. The tool does not know your site's actual paths; it only tests what you type.
Common questions
Does letter case matter here too?
Yes. /Shop/ does not match /shop/.
Does Allow follow the same logic?
Yes, Allow also matches as a prefix, and the longest matching rule wins.
On a hosted platform, can I change the rule?
Usually not. Use the tester to understand the platform's file, and contact its support if you find a problem.