Remove duplicate lines from a list

Paste a list and drop the repeated lines. Nothing leaves your browser.

How it works

  1. Paste your lines
  2. Choose how lines are matched
  3. Copy the list without repeats

What counts as a duplicate

Two lines are duplicates when they are the same text, character for character. By default the match is case sensitive, so Apple and apple are two different lines and both stay. Turn case sensitive off and they match. Ignoring case compares the lower-cased text (with the Greek final sigma ς treated as σ). It is not full Unicode case folding, so a few letters, such as the long s ſ and s, still count as different. Turn on ignore spaces at line ends and cherry matches a line that is cherry with spaces or tabs before or after it. Only the comparison ignores those spaces. The line that is kept is copied exactly as you typed it, spaces included. Spaces inside a line always count, so red apple and red apple (two spaces) are different lines. Accents count too, so cafe and café are different.

Which copy is kept

With First selected, each line stays where it first appeared and later repeats are removed, so the order of your list is unchanged. With Last selected, each line stays at the position of its final appearance, and earlier copies are removed. The two settings give the same set of lines but can give a different order. When case or spaces are ignored, the kept copy decides what you see: a list of apple and Apple gives apple with First and Apple with Last. Sorting is off unless you turn it on. When it is on, the kept lines are put in A to Z order, and the choice of First or Last only decides which spelling survives.

Blank lines and line endings

A blank line is a line like any other. The first blank line is kept and later ones are removed, so a list with five empty lines ends up with one. If you want no blank lines at all, use a blank line remover after this tool. A line with only spaces is blank for matching only when ignore spaces at line ends is on. Windows line endings, old Mac line endings and Unix line endings are all read as one line break each, and the result uses a single plain line break. A trailing empty line at the end of your text counts as a line and is treated the same way.

Counts and limits

The result shows how many lines went in, how many came out and how many duplicates were removed. The last number is always the first minus the second. The work happens in your browser, in this tab. No server receives your list, and the list is not saved or put in the page address. Only your choices, such as keeping the last copy, are remembered in this browser. Text up to 1,000,000 characters is handled. Longer text shows a message asking you to split it into parts. Very long lists follow your typing after a short pause instead of on every key.

Frequently asked questions

Does it keep the original order?

Yes, unless you turn sorting on. With First selected, every line stays where it first appeared. With Last selected, every line stays where it last appeared, so the order can differ from your original list.

Is Apple the same as apple?

Not by default. The match is case sensitive until you turn that option off. With it off, Apple and apple count as the same line, and the copy you chose to keep (first or last) is the one that stays.

Why are two lines that look the same both still there?

They differ in a way that is hard to see. The usual cause is a space or tab at the end of one line. Turn on ignore spaces at line ends. Other causes are two spaces instead of one, a different accent, or a letter from another alphabet that looks the same.

What happens to blank lines?

The first blank line is kept and the rest are removed, because they are repeats of each other. To remove every blank line, run the result through a blank line remover.

Is my list uploaded?

No. The lines are processed in your browser, in this tab, and no server receives them. Your settings are saved in this browser, but the text is not, so closing the tab clears it. Google Analytics counts visits, results and copy presses, never the text.