Should I use the sorter's remove-duplicates option or the duplicate line remover?
If the repeats in your list are identical character for character, the sorter's «Remove duplicate lines» option is enough and gives you a sorted, clean list in one step. If the copies differ by diacritics or a trailing space, or you want to see what repeated and how often, start with the duplicate line remover and sort afterwards.
What the sorter's option actually does
The option in the Line sorter removes any line that matches an earlier one, then sorts what is left. Matching is literal with two exceptions: English letter case is ignored unless you turn on «Case-sensitive», and spaces at the start of a line are skipped if you turned that option on. Nothing more.
I tried a six-line list: قهوة, شاي, قهوة, قَهوة (with a vowel mark), تمر, and «شاي » with a trailing space. The option removed one line only, the second قهوة, and returned تمر, شاي, «شاي », قهوة, قَهوة. Five lines stayed, including two versions of tea and two of coffee that any reader would call duplicates.
What the duplicate line remover adds
The Duplicate-line remover can match while ignoring diacritics and tatweel, ignore spaces at both ends of a line, and handle empty lines three ways: like any line, keep them all, or remove them all. It does not sort. It keeps the first occurrence of each line in place, in the original order.
The same list with ignore diacritics and ignore spaces turned on came down to three lines: قهوة, شاي, تمر, with three removed. The tool also lists what repeated: قهوة appeared 3 times with 2 removed, شاي twice with 1 removed. That table is handy when you clean a customer list and want to know who signed up more than once.
When the sorter alone is enough
- The list came out of a single system, so repeats are exact copies.
- You want the result sorted anyway and the original order does not matter.
- The only difference is English letter case, like Coffee and coffee: the sorter treats them as one line by default, while the duplicate remover is case-sensitive by default and needs that switched off.
When to start with the duplicate remover
- The text was copied from different sources, some with vowel marks or stretched letters.
- You suspect stray spaces at the end of lines.
- You need to see what repeated and how many times, not just make it disappear.
- You want to keep the original order, such as the order orders arrived in.
Combining them is easy: clean the list in the duplicate remover, then paste the result into the sorter. In our example that gave the final order تمر, شاي, قهوة.
A note on which copy survives
In both tools the first occurrence stays exactly as written. If the first copy is the one with a vowel mark or a stray space, that is the version you get. Check the top of your list if the spelling matters to you.
Common questions
Does either tool treat «أحمد» and «احمد» as the same?
No. Neither unifies hamza forms or ta marbuta, so those are two different lines in both tools.
How do I see how many lines went?
The sorter prints the number of duplicate lines removed under the result; the duplicate remover shows the count plus a table with the details.