Duplicate Line Remover

Fuzzy matching catches what exact matching misses

The reason to use this one rather than any of the dozens of one-button versions.

Exact matching only removes lines that are character-for-character identical. Real lists are not like that. The same company appears as Acme Ltd and Acme Ltd. The same person is Jon Smith and John Smith. The same keyword has a stray letter in it. Every one of those survives an exact dedupe, and they are usually exactly what you were trying to remove.

Fuzzy matching compares how similar two lines are rather than whether they are identical, and treats anything 80% alike or closer as the same entry. That catches a trailing full stop, a single typo, a missing letter, a different spelling of the same name.

But never use fuzzy on numbers or codes

The trade-off, and it is worth being precise about because the failure is silent.

Similarity is measured against the length of the line, so short strings are dangerous. Consider two order numbers: 10001 and 10002. They differ by one character out of five, which makes them exactly 80% similar — so fuzzy matching treats them as the same entry and deletes one of them.

The same applies to SKUs, postcodes, reference numbers, version strings, part codes and dates. Anything where a one-character difference is the entire point.

So: fuzzy for human-written text — names, companies, addresses, keywords, titles. Exact for anything machine-generated. And if a list has both, run it exact, then check the result by eye.

One practical note: fuzzy matching compares every line against every line it has kept, so a long list takes noticeably longer than an exact match on the same data. On a few hundred lines it is instant; on tens of thousands, expect to wait.

If duplicates survive, the problem is almost always invisible

The single most common complaint about tools like this, and the cause is nearly always the same.

Two lines that look identical on screen are not identical if one ends in a space. A trailing space, a tab, or the carriage return that Windows text files carry are all invisible and all make an exact match fail.

Which is why Trim whitespace is switched on by default here — it strips leading and trailing space before comparing, so lines that look the same are treated the same. Leave it on unless you specifically need to distinguish indented from indented, which is occasionally true for code or structured text.

Case sensitivity is off by default, so Apple and apple count as one entry. Turn it on when case carries meaning — variable names, case-sensitive identifiers, or a list where IT and it are different things.

Turn it around: show the duplicates instead

The mode worth knowing about, because it answers a different question.

Unique only is the default and gives you the cleaned list. Duplicates only shows you just the lines that appeared more than once — which is what you want when you are investigating rather than cleaning. Which email addresses were entered twice? Which SKUs are repeated across two exports? Which keyword is in the list three times?

Both with count keeps the full deduplicated list and marks each repeated line with how many times it appeared. That turns the tool into a frequency tally: paste a column of survey answers, categories or tags, and read off which values are most common.

The stats line under the boxes always tells you how many lines went in, how many came out, and how many duplicates were removed — which is the quickest sanity check that the settings did what you expected.

Keep the first or the last one

This matters whenever the order of your list means something.

First occurrence is the default and right when the list is in priority order, or when the earliest entry is the authoritative one.

Last occurrence is right when later entries supersede earlier ones. A log where the newest line is the current state. An export appended to over time. A contact list where the most recent address is the correct one. Getting this backwards is a quiet way to keep the wrong data, so it is worth a thought rather than a default.

The sort options — A–Z, Z–A or by length — apply to the output afterwards. Sorting by length is unexpectedly useful for spotting problems: truncated entries and stray fragments collect at one end of the list where you can see them.

Nothing is uploaded

All of it happens in your browser — no file is sent anywhere, and there is nothing to be stored or deleted later. Worth having, given what these lists usually are: email addresses, customer names, order references, exported database columns, keyword research. A list of a few thousand email addresses pasted into a random web tool is a data protection problem; pasted into this one, it never leaves the tab.

Related tools

Line Break Remover handles the other common list problem — text arriving with breaks in the wrong places. Word Counter counts and analyses a block of text, and CSV to JSON Converter is the route onward when your cleaned list needs to become structured data.