How to remove duplicate lines from a list

The first occurrence stays, in its place

The tool keeps the first occurrence of each line exactly as you wrote it and in its original position, and removes the repeats after it. So your list keeps its order. It also tells you how many lines it removed and which lines repeated.

Matching options

• Case sensitivity: on by default.
• Ignore spaces at the start and end of a line.
• Ignore Arabic diacritics when comparing, so a vocalized word and the bare word count as one line, while the kept line stays as you wrote it.
• Empty lines: treat them like any line, keep them all, or remove them all.

A practical case: you merged two customer lists, one from your store and one from an exhibition, to send them an offer. Without removing repeats, some customers get the same offer twice, which can annoy them enough to unsubscribe. A minute of cleaning before sending prevents that.

How to do it

1. Open the Duplicate-line remover and paste your list, one item per line.
2. Choose the matching options.
3. Check the result and the number of removed lines, and open the list of repeats if you want to verify.
4. Copy the result or download it as a text file.

What to know

• Matching is literal: «أحمد» and «احمد» are different lines, and so are the Arabic-Indic digits ٠ to ٩ and the digits 0 to 9.
• The ignore-diacritics option affects the comparison only, and never changes the line that is kept.
• Tested on 100,000 lines, and the work is done in under a second.

After removing repeats, sort the list alphabetically with the Line sorter. If extra spaces stop repeats from being detected, clean them first with the Whitespace cleaner.