How do I ask AI crawlers such as GPTBot not to collect my site's content, without blocking search engines?
Add a separate group for each crawler you want to keep out, named exactly as its owner publishes it in their documentation, with Disallow: /. Leave your * group alone. A crawler named in a group follows that group only, while everything else, search engine crawlers included, follows the * group. Remember, though: this is a request, and only crawlers that choose to respect robots.txt will follow it.
Why a separate group
Under RFC 9309, a crawler looks for a group carrying its own name, and if it finds one it follows that group alone and ignores the * group. That is what makes this approach safe: a group for GPTBot that blocks everything leaves the * group, which Googlebot and others follow, untouched.
The name must match what the crawler sends in its User-agent header, and the tool compares it case-insensitively. We cannot give you a complete list of AI crawler names; they change and new ones appear. Take each name from the official documentation of the company that runs the crawler.
Steps in the tool
- In the Robots.txt builder, keep your current group for all crawlers (*) as it is, for example blocking /cart/.
- Press 'Add a group' and type the crawler name, such as GPTBot, into User-agent.
- Type a single / into Disallow, meaning the whole site.
- For several crawlers with the same rules, put their names in the same group separated by commas. The tool writes a separate User-agent line for each name.
- Order the groups however you like; order does not change the result.
This is what came out with a GPTBot group and a * group blocking /cart/:
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /cart/
Test before publishing
In 'Test a path' we entered /products/oud twice:
- With GPTBot: blocked from crawling, winning rule Disallow: /, using its own group. The same result appeared when we typed the name in lower case as gptbot.
- With Googlebot: allowed to crawl, no rule matches, and the * group was used because no group is named Googlebot.
Note that /cart/ is not in the GPTBot group, which does not matter here since Disallow: / blocks everything anyway. If you want partial rules for a named crawler, remember it inherits nothing from the * group, so write everything it needs into its own group.
Limits to know
- The file is a request, not a block. A crawler that ignores robots.txt is unaffected; stopping it needs a server or firewall rule.
- The rules do not delete anything collected before. They concern crawling from now on.
- The file is public, so anyone can see which crawlers you asked to stay away.
- On hosted stores you usually do not control the file. Check what your platform publishes and ask its support what options exist.
Common questions
Does this affect how I appear on Google?
The separate group does not change Googlebot's rules, and the tester confirms it. We make no promises about search results, though.
Do I include the crawler's version number?
Write the name alone. The tool rejects a name containing a space or a colon.
Do I still need a Sitemap line?
Not for this, but it helps search engines if you have a sitemap.