I have a test copy of my site on a subdomain. Is blocking it entirely in robots.txt enough? And what is the risk when it goes live?
It is not enough. A full block asks crawlers that respect the file to keep away, but it hides the site from nobody, and its links can still show up in results if other pages point to them. Protect the test copy with a password. The bigger risk comes at launch: the block-everything file travelling with the rest of the files to the real site.
What a full block does and does not do
In the Robots.txt builder, the 'Block everything' preset writes this file:
User-agent: *
Disallow: /
It comes with a blunt warning: the rule / blocks the whole site for every crawler. The tool shows that warning whenever the * group blocks the whole site, whether it came from the preset or you typed it. The preset's own description says it suits only a staging or test copy, and that a login is the real protection even there.
- It does: ask well-behaved search engine crawlers not to fetch pages.
- It does not: stop anyone from opening the site, or stop crawlers that ignore the file.
- It does not: guarantee the links stay out of results. A URL other pages link to can appear without a description, because the crawler cannot read the page to see anything on it.
- It reveals: the file is public, so anyone opening /robots.txt on the subdomain learns there is a copy you want kept quiet.
Real protection for a test copy
- Put a password on the whole subdomain from your hosting panel, or limit access to your team's IP addresses.
- Keep the block-everything file on staging as an extra layer, not the only one.
- Do not share staging links in public messages or on pages of the live site.
- Do not make a sitemap for staging, and never list its links in the live site's sitemap.
The risk on launch day
The classic mistake is a developer copying the whole staging folder to the live site, robots.txt included. On a live site, as the tool's warning puts it, that means disappearing from search results over time, with pages already indexed lingering without a description. Nobody may notice for weeks, because the site works normally for visitors.
A short launch-day checklist:
- Open the live address followed by /robots.txt right after launch and read it with your own eyes.
- Build the live file afresh in the tool, for example from 'Allow everything' or 'Block specific folders', and add a Sitemap line pointing to the sitemap you made in the XML sitemap generator.
- In 'Test a path', test the home page and a product page with Googlebot and confirm both are allowed.
- If deployment is automated, keep a separate robots.txt per environment rather than copying one.
Common questions
Does the subdomain need its own file?
Yes. Rules apply only to the host the file came from, so the subdomain's file does not affect the main site, and the other way round.
The test copy is already in results. Will blocking remove it?
No. Blocking stops the crawler seeing any noindex tag. Password-protect it and request removal in your webmaster tools.
Can the tool tell whether my live file blocks everything?
No, it does not read your current file. Copy its rules into the tool's groups to test them.