I copied SKU codes from Excel and there are obvious repeats, but the tool doesn't remove them; what's the cause?
Most often an invisible space before or after the code in some cells, which makes 'SKU-1002' with a trailing space a different line from 'SKU-1002'. Turn on 'Ignore spaces at the start and end of a line when matching' and the codes match and the repeats go. The tool keeps the first occurrence with its spaces, so clean the list with the whitespace tool if you want tidy codes.
A worked example
A column of four codes copied from your stock file, with two obvious repeats:
Line 1 is SKU-1001, line 2 is SKU-1002 followed by a space, line 3 is SKU-1002, line 4 is SKU-1001 preceded by a space.
Default settings: 4 input lines, 4 kept, 0 removed, and the message 'No repeated lines with these settings'.
With spaces at both ends ignored: 4 input lines, 2 kept, 2 removed.
The two kept lines are SKU-1001 and SKU-1002 as they first appeared, which means the second one still carries its trailing space.
Why Excel adds spaces
A space gets into a cell when it is typed by hand, pasted from another system, or exported by point-of-sale software with fixed-width fields. The cell looks normal in the sheet, and when you copy the whole column the space travels with it. A tab at the end of a line can come along too, and the same option ignores it: we tried SKU-1002 followed by a tab and it counted as a repeat.
Steps
- Copy the code column from Excel and paste it into the Duplicate-line remover.
- Tick 'Ignore spaces at the start and end of a line when matching'.
- Check 'Lines that repeated': every repeated code with how many times it appeared.
- Copy the result. If it is going back into your stock system, clean it first as described below.
Why the space is still in the result
Because the tool never edits lines; it only decides which ones stay. Ignoring applies to matching only, and the first line stays as you wrote it. For a product code that matters, because 'SKU-1002' with a space may not match the same code in your system.
The fix is to clean before removing: paste the column into the Whitespace cleaner with its default rules. In our example it trims one space from the start of a line and one from the end of another, and hands you clean codes. After that you do not even need the ignore option: removing duplicates with the defaults gives 2 kept and 2 removed, and both codes are free of spaces.
Cases the both-ends option does not handle
- A space inside the code: 'SKU 1002' and 'SKU 1002' with two spaces stay separate lines, because the option never touches the middle of a line.
- A zero-width space at the end of a code: it is not an ordinary space, so the option does not ignore it. We tried it and both codes stayed. The whitespace cleaner removes it if you turn on its zero-width rule.
- Different letter case, such as sku-1002 and SKU-1002: that is a job for the case-sensitivity option, not the spaces option.
A non-breaking space copied from a web page, on the other hand, is ignored by the both-ends option when it sits at the start or end of the code.
Common questions
Does the order of codes change?
No, the first occurrence of each code stays in its place, and you can sort them later.
Can it handle thousands of codes?
Yes, it was tested on 100,000 lines and computed the result in under a second.
Is my stock file sent anywhere?
No, everything stays in your browser.