What is the difference between a sitemap and a robots.txt file, and does my small site need both?
A sitemap lists the pages you want search engines to find; robots.txt asks crawlers not to fetch certain paths and can also say where the sitemap is. A small site can get by without either, but a sitemap helps new pages get discovered, and one line in robots.txt tells every crawler where it lives.
Two files pointing in opposite directions
- The sitemap (sitemap.xml) says 'here are my pages, come in'. An XML file with the link to each page you want found, and perhaps its last-modified date.
- robots.txt says 'please do not go here'. A text file with rule groups per crawler, such as Disallow: /cart/.
- Both are public, readable by anyone at /sitemap.xml and /robots.txt, and neither protects anything or guarantees indexing.
The usual mix-up is thinking robots.txt gets pages into search, or that a sitemap keeps them out. Neither is true: a sitemap only helps discovery, and a block stops crawling, not indexing.
Does your small site need them?
A five-page site with good internal links may be found without any file. But when you keep adding pages, such as a blog or seasonal offer pages, a sitemap makes them easier to discover. robots.txt is unnecessary if there is nothing you want crawlers to skip, such as the cart, checkout or internal search results.
If your store runs on Salla, Zid or Shopify, the platform usually serves both files already, and you generally cannot replace them. Open your store address followed by /robots.txt and then /sitemap.xml to see what is there.
Example: a two-page site
For a company site with a home page and an About page, we pasted the two links into the XML sitemap generator and got a single sitemap.xml with two URLs:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
</url>
<url>
<loc>https://example.com/about</loc>
</url>
</urlset>
Then in the Robots.txt builder we wrote a group for all crawlers blocking /cart/ and /checkout/, and entered the sitemap address in the Sitemap field. The tool wrote it at the end of the file:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://example.com/sitemap.xml
Steps
- List the pages you want in search and build the sitemap from them.
- Upload sitemap.xml to the site's root folder.
- Build robots.txt with the full sitemap URL in it, and upload it to the root too.
- Make sure robots.txt does not block any URL you listed in the sitemap; those two messages contradict each other.
Neither tool visits your site: the first converts the list you give it, the second builds the file from the rules you type. And the Sitemap line in robots.txt applies to the whole file, not to one group.
Common questions
Which should I start with?
If there is nothing to keep crawlers away from, the sitemap first. robots.txt comes in when you have something to block, or to point to the sitemap.
Is the Sitemap line enough instead of submitting in webmaster tools?
They are two ways of announcing the file. Check each search engine's documentation for what it prefers.
Will a sitemap make my pages appear faster?
We make no such promise. It helps discovery; indexing is the search engine's decision.