Remove Duplicate Lines
Clean any list by stripping repeated lines, with control over case sensitivity, whitespace, empty lines and sort order.
About the Remove Duplicate Lines
Duplicate lines accumulate wherever lists are merged. Two email exports become one file with hundreds of repeats. A keyword list assembled from three research tools contains the same term in slightly different casing. A log file repeats an identical error a thousand times and hides the three interesting lines in between. Spreadsheets can do this, but opening a spreadsheet, pasting, running a formula and copying back out is a five-minute job for a five-second problem.
This tool removes duplicates from a pasted list and gives you the controls that actually matter. Case-insensitive matching treats Newsletter and newsletter as the same entry, which is almost always what you want for email addresses and tags. Trimming whitespace before comparing catches the extremely common case where one copy of a line has a trailing space and therefore looks unique to a naive comparison. Removing empty lines cleans up the gaps that copy-and-paste leaves behind.
There is also an inverted mode. Instead of keeping one copy of everything, you can keep only the lines that appeared more than once, or only the lines that appeared exactly once. The first is how you find which entries two lists have in common; the second is how you find the entries that are unique to a single source. Both are questions people normally reach for a spreadsheet formula to answer.
How to use it
- Paste your listOne item per line — emails, URLs, keywords, SKUs, names or log lines. There is no row limit.
- Set the matching rulesChoose whether comparison ignores case and whether surrounding whitespace is trimmed before lines are compared.
- Pick what to keepKeep the first occurrence of each line, keep only lines that appear once, or keep only lines that were duplicated.
- Optionally sort and cleanSort the result alphabetically or numerically, and drop empty lines in the same pass.
- Review the summaryThe tool reports how many lines went in, how many duplicates were removed and how many remain.
- Copy or downloadTake the cleaned list back to your spreadsheet or save it as a .txt file.
Features
- Case-insensitive optionTreat differently-cased versions of the same entry as duplicates — essential for email and tag lists.
- Whitespace trimmingCompare lines after stripping leading and trailing spaces, catching the most common false-unique.
- Three keep modesUnique-first, only-once and only-duplicated cover deduplication, difference and intersection tasks.
- Optional sortingAlphabetical, reverse alphabetical or numeric sorting applied after deduplication.
- Empty line removalClear the blank rows that copy-and-paste between applications leaves behind.
- Full statisticsInput count, duplicates removed and output count reported after every run.
Who uses this tool
- Email marketersMerge subscriber exports from several sources and remove the overlap before an import, avoiding duplicate sends.
- SEO researchersConsolidate keyword lists from multiple tools into one clean set without paying for a deduplication feature.
- DevelopersCollapse a repetitive log file down to its distinct messages so the unusual entries become visible.
- Anyone with a spreadsheetGet a distinct list without writing a UNIQUE formula, building a pivot table or installing an add-in.
Tips worth knowing
- Turn on both “ignore case” and “trim whitespace” for email lists — those two options together catch the overwhelming majority of real-world duplicates.
- To find what two lists have in common, paste both one after the other and choose “keep only duplicated lines”.
- Deduplicate before sorting when you care about preserving the original order of first appearance; the tool always keeps the first occurrence.
Frequently asked questions
Does the tool keep the first or the last copy of a duplicate?
The first. If the same line appears at positions 3, 40 and 92, the copy at position 3 is kept and the others are discarded, so the original ordering of your list is preserved as closely as possible.
How do I find entries that appear in both of two lists?
Paste the first list, then the second list directly underneath it, and select “keep only duplicated lines”. What comes back is the intersection — every entry that appeared in both sources.
Why are two lines that look identical not being matched?
Almost always invisible whitespace: a trailing space, a tab, or a non-breaking space pasted from a web page. Enable “trim whitespace” and they will match. Different Unicode representations of the same accented character can also cause this, which is rarer but does happen with data from older systems.
Is there a limit on list size?
No fixed limit. Deduplication uses a hash set, so it scales close to linearly — a list of 100,000 lines processes in well under a second in a modern browser.
Can I remove duplicate words rather than lines?
Put each word on its own line first. If your words are comma-separated, use find-and-replace in any editor to swap commas for line breaks, deduplicate here, then reverse the swap.
Is my list uploaded anywhere?
No. All processing happens in your browser. This matters for the common case of deduplicating a customer or subscriber list, which is personal data you should not be pasting into an unknown server.