Remove Duplicate Words
- 3
- Words removed
- 43
- Characters out
Duplicate words are removed from text, with the important choice between two very different jobs: deleting only words repeated back to back, the "the the" that survives editing, or deleting every repeat of a word anywhere in the text, which produces a vocabulary list rather than readable prose.
How it works
Words are compared on their letters and digits only, so "cat," and "cat" count as the same word and the punctuation stays attached to whichever copy is kept. Comparison is case-insensitive by default, which catches "The the" at the start of a sentence.
- Back-to-back only: the safe mode. It fixes the classic editing artefact and leaves the meaning of the text intact.
- Every repeat anywhere: destroys readable prose, because the second "the" in a sentence is usually necessary. It is for building a distinct-word list, not for editing.
- Restarting at each line break keeps the comparison inside a line, which is what you want for a list where each line stands alone.
- When a word is removed, its preceding space goes with it, so the result has no double spaces where the repeat was.
Examples
The classic editing mistake
Text
the the quick brown fox fox jumps over the lazy dog dog
Result
the quick brown fox jumps over the lazy dog: 3 removed
Back-to-back mode. The second "the" later in the sentence survives, because it is a different occurrence doing a different job.
Every repeat, anywhere
Text
the quick brown fox jumps over the lazy dog
Result
the quick brown fox jumps over lazy dog
The second "the" is deleted and the sentence is now wrong. This mode is for extracting distinct words, not for editing prose.
Punctuation between repeats
Text
Wait, wait: I know
Result
Wait,: I know
The comma is not part of the word, so "Wait," and "wait" match. The first copy keeps its punctuation.
Frequently asked questions
Which mode should I use for proofreading?
Back-to-back only. It removes the accidental "the the" that editing leaves behind without touching legitimate repetition. The every-repeat mode will silently break sentences that need a word more than once.
Why did it remove a word I needed?
You are almost certainly in "every repeat, anywhere" mode, which deletes the second occurrence of any word regardless of whether the sentence needs it. Switch to back-to-back only.
Does punctuation stop two words from matching?
No. Comparison uses only the letters and digits, so "cat," and "cat" are the same word. The punctuation stays on the copy that is kept, so the text stays properly punctuated.
How is this different from removing duplicate lines?
That tool compares whole lines and removes the repeated ones. This works inside lines, word by word. A line-based tool will not notice "the the" at all, since it only ever compares the full line.
Can it tell me which words repeat without removing them?
Not directly. The keyword frequency counter is the tool for that. It lists every word with its count, which is what you want when the repetition itself is the thing you are investigating.