I pasted 120 links but the sitemap came out with only 97. Where did the rest go?
Into one of three places, all shown by the tool: duplicates removed, links that differed from another only by a part after # and became duplicates once it was dropped, and rejected lines that were not valid full URLs. Blank lines are not counted at all. Add up the numbers in the summary and you will reach your total.
A small example that shows the arithmetic
We pasted five lines:
https://example.com/a
https://example.com/a#reviews
https://example.com/a
/b
https://example.com/c
The file came out with just two URLs, page a and page c. This is what the summary said:
- Duplicates removed (2): line 2 duplicate of line 1, and line 3 duplicate of line 1.
- A note: the #fragment was removed from one URL, because search engines ignore it. That is what made line 2 a duplicate; without #reviews it is page a itself.
- Rejected lines (1): line 4, because /b is not a full URL.
The sum: five lines, minus two duplicates, minus one rejected line, equals two URLs. The same way, if you pasted 120 lines and got 97, duplicates plus rejected plus blank lines should come to 23.
Why the part after # is dropped
The part after # sends the browser to a spot inside the same page, such as the reviews section. It does not make a new page, so the tool removes it. If you copied links from buttons inside your pages, you may find many like that, and they collapse to one URL per page. That is the intended result.
What the tool does not count as a duplicate
We also tried four forms of one link. A link differing only in the case of the host, such as Example.com, counted as a duplicate, because the tool lower-cases the host. But the link ending in a slash, /a/, and the one with a capital in the path, /A, stayed as separate URLs from /a. On a server those can genuinely be different pages, so the tool does not merge them. If you have one page, delete the form your site does not use.
How to review the result
- In the XML sitemap generator, look at the counts above the output: valid URLs, duplicates removed, rejected lines.
- Open 'Duplicates removed' to see each line number and the earlier line it repeats.
- Open 'Rejected lines'; these are the only ones that may be real pages you lost. Fix them and generate again.
- If the list mixed hosts and you chose one, remember the other host's URLs were left out too.
To see and clean repeats before pasting, run the list through the Duplicate-line remover. And remember the tool never visits your site, so a URL that stayed in the file is not necessarily a page that works.
Common questions
Does the tool remove broken links?
No. It opens no links, so it cannot know whether one works.
Are blank lines counted as rejected?
No, they are ignored and do not appear in any list.
How many duplicates does the list show?
The first hundred, with a count of the rest.