Search Console says a product page is blocked by robots.txt. How do I find out which rule is blocking it?

Why reading the file by eye is not enough

With twenty rules in a file, some with stars and some with a $, it is hard to see which one applies to a given link. The winning rule is not the first in the list; it is the longest of the matching rules, and Allow wins a tie of equal length. The tester applies exactly that calculation and gives you the answer.

Steps

1. Open your site address followed by /robots.txt and copy each group into the Robots.txt builder: names into User-agent, paths into Disallow and Allow.
2. Copy the page's full URL from the Search Console report.
3. Paste it into the path field under 'Test a path', and set the crawler the report concerns, such as Googlebot.
4. Read the verdict: blocked or allowed, the winning rule, and the group used.

Example: a broad rule catching variants

In a file containing Disallow: /*? to stop sort links, we pasted https://example.com/products/oud?variant=123. The tool took the path /products/oud?variant=123 from it and noted that only the path was tested, because robots.txt rules never look at the domain. The verdict: blocked from crawling, winning rule Disallow: /*?.

The pattern /*? means 'any path containing a question mark'. Its author wanted to block sorting and blocked every variant link along with it. The bare product page, /products/oud, came back allowed, which is why nobody may notice until the report flags it.

Two ways to fix it

• Narrow the rule to the parameters you really mean, such as /*?sort=. We tested the same link against that rule and it became allowed, because it no longer matches.
• Or keep the broad rule and add the exception Allow: /*?variant=. It is longer than /*?, so it wins; the link came back allowed with Allow: /*?variant= as the winning rule.

User-agent: * Disallow: /*? Allow: /*?variant=

Before choosing, ask whether you want variant links crawled at all. If they show the same product, leaving them blocked may be fine and the report is just a notice. The decision is yours; the tool shows the effect of each option.

After the fix

The tool neither uploads the file nor reads your live one. Download the new file, upload it to the domain root, then open /robots.txt in a browser to confirm. The report can take time to update, and some tools let you request a recheck; see their documentation. Once the page is allowed, look at how its title and description might appear in the Search snippet preview. On a hosted store you usually cannot edit the file yourself, so send what you found to the platform's support.