Why does the tool warn me when I block the /assets/ folder, even though it holds no pages for visitors?
Because that folder usually holds the CSS and JavaScript a page needs to look the way visitors see it. Search engines render pages much like a browser, so if those files are off limits they may see a broken or empty page and misjudge its content. The tool tests common asset paths against your rules and warns when one is blocked. Remove the rule, or add Allow lines for the CSS and JS folders.
What the tool actually checked
The tool does not know what is in your folders, since it never visits your site. Instead it carries a short list of common stylesheet and script paths, such as /assets/css/main.css, /js/app.js and /static/main.js, and tests each rule group against them. If any is blocked, the warning names the rule and the sample path.
We wrote a group for all crawlers containing only Disallow: /assets/, and the warning appeared: the rule /assets/ blocks CSS or JavaScript files, for example /assets/css/main.css. The file itself was still produced, so the warning does not stop you; it asks you to think.
Why crawlers need these files
A modern product page may only show its images, price or description after JavaScript runs, and its whole layout comes from CSS. A crawler that sees the HTML alone, without those files, may see something very different from what your customer sees. We cannot say exactly how each search engine handles that; check its documentation on rendering.
The fix: an exception, not a deletion
If /assets/ holds something you do want blocked, such as staff-only files, keep the rule and add exceptions for styles and scripts:
User-agent: *
Disallow: /assets/
Allow: /assets/css/
Allow: /assets/js/
With this file the warning disappeared. We tested two paths: /assets/css/main.css became allowed, with Allow: /assets/css/ as the winning rule because it is longer than /assets/. /assets/private/price-list.pdf stayed blocked by Disallow: /assets/.
- Write your rule in the Robots.txt builder and read the warnings under the file.
- If the CSS or JavaScript warning appears, open the source of one of your pages and look for .css and .js paths to learn their real folders.
- Add an Allow line for each of those folders inside the same group.
- Test the path of a real file from your site in 'Test a path'.
Limits of the check
The list the tool checks is a sample of common paths, not your site's paths. If your files live in a differently named folder such as /theme/build/, no warning will appear even if you block it. That is why step four matters: test your own paths. And remember robots.txt is no way to hide a private file; it is public, and a private file needs protection on the server. Once you are sure the page renders in full, check its title and description in the Search snippet preview.
Common questions
Is the warning shown per group?
Yes, the tool checks each group, except a group that blocks the whole site, which gets its own warning.
Should I block font and image files too?
The tool checks CSS and JS only. If you want your images to appear in search, do not block their folder.
Are files on another host affected by my file?
No. Files on another host, such as a content delivery server, follow that host's own robots.txt.