Robots.txt builder: questions and guides
Each guide here answers one question with a practical example, then takes you to the tool to try it yourself. Guides: 11.
Questions
- robots.txt: what it blocks and what it doesn'tHow to write a robots.txt that asks search engines not to crawl cart and account pages, and why the file hides nothing and does not remove a page from results.
- My store is on Salla or Zid and I cannot edit robots.txt. How do I check that my product pages are not blocked by it?On a hosted store you read the file rather than write it. Copy its rules into the builder, test real product paths and see which rule decides.
- How do I ask AI crawlers such as GPTBot not to collect my site's content, without blocking search engines?A separate group per named crawler with Disallow: /, leaving the * group for search engines as it is. How to test it, and why only crawlers that choose to comply obey.
- Search Console says a product page is blocked by robots.txt. How do I find out which rule is blocking it?Rebuild your rules in the builder, paste the report's full URL into the tester and see the winning rule. Why a pattern like /*? catches variant links.
- I wrote Disallow: /shop to block one folder. Why is /shop-by-brand blocked too?A robots.txt rule matches every path that begins with it, so a rule with no trailing slash catches more. How to limit it to the folder, and block the page with $.
- My category pages repeat under ?sort= and ?filter= links. Should I block them in robots.txt, and how do I write the rule so nothing slips through?/*?sort= only matches when sort comes right after the ?, so ?page=2&sort=price slips through. How to catch both forms, and when blocking parameters is worth it.
- Why does the tool warn me when I block the /assets/ folder, even though it holds no pages for visitors?Crawlers render pages much like a browser, so with stylesheets and scripts blocked they may see a broken page. How the builder spots it, and how Allow rules fix it.
- I have a test copy of my site on a subdomain. Is blocking it entirely in robots.txt enough? And what is the risk when it goes live?Disallow: / asks polite crawlers to stay away but hides nothing, and links can still surface in results. Use a password, and keep that file off the live site.
- I added a group just for Googlebot and now it ignores the rules in my general * group. Why?A crawler follows the one group that names it and drops the * group entirely, so groups do not stack. How the tester shows it, and how to copy shared rules across.
- My blog is on a subdomain like blog.mystore.com. Why do the rules in the main domain's robots.txt not apply to it?Crawlers look for robots.txt at the root of each host separately, so a subdomain reads its own file. How to build one for the blog, and why the tester ignores the domain.
- How do I write a robots.txt rule that blocks a folder whose name is in Arabic?Crawlers request Arabic paths percent-encoded, and the builder compares text exactly without decoding. Write the rule encoded and test it with a full URL.