Duplicate-line remover
Paste a list of products, keywords or email addresses and get it back without repeats. The first occurrence of each line stays exactly as you wrote it and in its original position, and the tool reports how many lines were removed and which ones repeated. It is a quick way to remove duplicate lines and de-duplicate a list before you move it into a spreadsheet or a shop.
Loading the tool…
Processing and privacy: The result is calculated inside your browser and your data is not sent to any server.
How to use it
- Paste your text into the input box, one item per line.
- Choose how lines are matched: case sensitivity, ignoring surrounding spaces and how empty lines are treated.
- Check the result and the number of removed lines; open the list of repeated lines if you want to verify.
- Copy the result or download it as a text file.
Example
The built-in example has 11 lines: تفاح, Apple, تفاح, apple, برتقال, “ Apple” with two leading spaces, تُفّاح with diacritics, an empty line, برتقال, an empty line, Banana.
With the default settings 8 lines stay and 3 are removed: the second “تفاح”, the second “برتقال” and the second empty line. If you turn off case sensitivity and turn on ignoring spaces and ignoring diacritics, only 5 lines stay: تفاح, Apple, برتقال, an empty line, Banana.
Limits and caveats
- Tested with 100,000 mixed lines (about 1.5 MB): the result is computed in well under a second, but the browser text box itself needs several seconds to display a paste of that size. For long results only the first 300 lines are shown in the preview; copy and download always contain the full result. Above about 4,000 lines or 150,000 characters the text box shows every line in one direction (taken from its first letter) instead of a direction per line, because per-line direction makes the browser extremely slow at that size.
- Matching is literal: Arabic alef forms (أ إ آ ا), ya and alef maqsura, ta marbuta and ha, Arabic-Indic digits (٠ to ٩) versus 0 to 9, and different Unicode encodings of the same letter are not unified.
- The diacritics option ignores harakat, shadda, sukun, tanwin, superscript alef, Quranic annotation marks and tatweel for comparison only. It never changes the line that is kept.
- An “empty” line is a line with no visible character, even if it contains spaces. The result is written with LF line breaks.
Questions about this tool
Which copy is kept when a line repeats?
The first one, exactly as written and in its place; later copies are removed. If ignoring spaces is on and the first occurrence is “ Apple” with two spaces, it keeps its spaces. Use the whitespace cleaner if you want them gone.
Are “تفاح” and “تُفّاح” the same line?
Not by default, because matching is literal and diacritics are extra characters. Turn on “ignore diacritics and tatweel” to treat them as one line; the first occurrence is the one kept.
Does the order of lines change?
No. The tool does not sort; it only removes repeats. Use the line sorter afterwards if you want sorted output.
How do I remove duplicate lines from a list?
Paste the list into the input box, one item per line, and the result appears without repeats together with the number of removed lines. For example, coffee, tea, coffee, milk, tea (one per line) becomes coffee, tea, milk, with 2 lines removed. Then copy the result or download it as a text file.
Can I remove duplicates regardless of upper and lower case?
Yes. Turn off “Case-sensitive matching” and Apple, apple and APPLE count as one line; the first occurrence is kept exactly as written. The option is on by default and has no effect on Arabic letters, which have no case. If lines differ only in spaces at the start or end, turn on the option that ignores those spaces too.