I added a group just for Googlebot and now it ignores the rules in my general * group. Why?

The example that exposes it

We built this file in the Robots.txt builder: a group for all crawlers blocking /admin/ and /cart/, then a Googlebot group blocking only /search, meant as an extra rule for it:

User-agent: * Disallow: /admin/ Disallow: /cart/ User-agent: Googlebot Disallow: /search

Then we tested the path /admin/login with two crawlers:

• Googlebot: allowed to crawl. No rule matches, and the group used is Googlebot. Its group does not mention /admin/, and it inherited nothing from *.
• Bingbot: blocked by /admin/. No group carries its name, so the * group was used on its own.

And /search?q=oud with Googlebot was blocked by its own rule, so the group works, but it works alone. The tool says as much when it uses the * group: the * group is used on its own and is not merged with anything else.

The fix: repeat the shared rules

Copy every rule you want for all crawlers into the Googlebot group as well:

User-agent: * Disallow: /admin/ Disallow: /cart/ User-agent: Googlebot Disallow: /admin/ Disallow: /cart/ Disallow: /search

1. In the Googlebot group, paste every Disallow and Allow line from the * group.
2. Add its own rule underneath.
3. In 'Test a path', try /admin/login with Googlebot again and confirm it is now blocked.
4. Try an ordinary path such as /products/oud too, to be sure nothing unintended is blocked.

Details worth knowing

• Names are compared case-insensitively, so a googlebot group applies to Googlebot.
• The tool takes only the leading name from a long user-agent string, so Googlebot/2.1 with the rest of its header is treated as Googlebot.
• A name that differs even slightly is another crawler. When we tested Googlebot-Image against the first file, the * group was used, because no group carries that full name. Real crawlers may behave differently in these details, so check each search engine's documentation about its crawlers.
• If you name the same crawler in two groups, the tool warns you, because crawlers merge groups with the same name and one group is clearer.

Sitemap lines are different: they apply to the whole file, not to a group, so there is no need to repeat them. Enter the address of the sitemap you made in the XML sitemap generator once in the Sitemap field. The tool does not read your current file, so copy its rules in before testing.