Remove Duplicate Lines vs Remove Duplicate Words
How they compare
| Criterion | Remove Duplicate Lines | Remove Duplicate Words |
|---|---|---|
| Unit compared | A whole line | A single word, with punctuation ignored in the comparison |
| Catches "the the quick fox" | No: it is one line, and one line cannot duplicate itself | Yes, and that is its main jobBetter here |
| Catches a row repeated in a list | YesBetter here | No: it never compares across lines as units |
| Order preservation | Preserved, with a choice of keeping the first or the last occurrence | Preserved: words are removed in place |
| Safe on prose | Yes, though it rarely finds anythingBetter here | Only in back-to-back mode. Removing every repeat anywhere breaks sentences |
Which is best for you
Deduplicate lines when you have a list
Email addresses exported twice, log lines repeated, a merged list with overlap. Order is preserved, which is the advantage over sorting to group duplicates together. Two options do most of the work: ignoring surrounding spaces catches the duplicate hiding behind a trailing space, and keeping the last occurrence is right when later entries supersede earlier ones.
Deduplicate words when proofreading
The classic "the the" that survives editing. Use back-to-back mode, which removes only words repeated immediately and leaves legitimate repetition alone. The every-repeat-anywhere mode exists for building a distinct-word list, and it will silently break prose. The second "the" in a sentence is almost always doing a job.
The recommendation
Pick by asking what your duplicate is: a whole row, or a word inside one. If a list has the same entry twice, that is the line tool and the word tool will not touch it. If a sentence says "the the", that is the word tool and the line tool cannot see it. When proofreading, stay in back-to-back mode, the alternative deletes the second occurrence of every word in the document, which produces something that reads like a telegram.
Frequently asked questions
Why did the line tool miss a duplicate I can see?
Almost always trailing whitespace or different capitalisation. Two lines that look identical on screen can be different strings. Turn on "ignore surrounding spaces" and turn off case sensitivity, between them they catch nearly every invisible difference.
Can I find duplicates without removing them?
The line tool lists every repeated line with its count alongside the result, so you can read that before copying anything. For words, the keyword frequency counter is the better tool. It shows every word with how often it appears.
Both tools
- Remove Duplicate LinesRemove duplicate lines from a list while keeping the original order, and see exactly which lines repeated and how often, rather than only getting the result.
- Remove Duplicate WordsRemove duplicate words either back to back, catching the repeated word that survives editing, or everywhere at once, with punctuation ignored when comparing.