Remove Duplicate Lines

When a list contains several copies of the same line, there has to be some rule for which copy survives, and keeping the first is the choice that preserves the most information about the original list: the surviving order still reflects however the list was originally assembled, rather than being scrambled by the deduplication step itself. This matters concretely for a list that was originally in a meaningful order — entries added chronologically, or already sorted by some external process — where deduplicating without disturbing that order keeps the result usable in the same way the original was, just without the repeats.

Turning on sorting afterwards is a separate, deliberate step precisely because it discards that original ordering in favour of alphabetical order, which is the right choice when the original order carried no meaning worth preserving — a jumbled export from several sources, for instance — and the wrong choice when it did.

"Ignore case" changes only how two lines are compared to decide whether they are duplicates — Apple, APPLE and apple are treated as the same entry under this setting, and whichever one appeared first in the list is the one that survives, complete with its own original capitalisation intact. This is a comparison-only setting, not a normalisation one: the tool never rewrites a surviving line's case to match some standard form, it simply decides which lines count as equivalent for the purpose of finding repeats.

This matters for lists gathered from several sources with inconsistent capitalisation — email addresses submitted through different forms, product names typed by different people — where without this setting, "John@Example.com" and "john@example.com" would be treated as two distinct entries and both would survive, defeating the purpose of deduplicating in the first place.

Inverting the operation to show only the lines that appeared more than once turns this from a cleanup tool into an auditing one — useful when the goal is not a clean list but an answer to "does this list actually contain duplicates, and which ones." This is the more useful mode when investigating a data quality problem: a customer list suspected of containing repeat sign-ups, a keyword list that may have been assembled from overlapping sources, a log file where repeated lines might indicate a genuine error being logged repeatedly rather than once.

Two lines that look visually identical can still fail to match as duplicates if one carries a trailing space or an extra tab character invisible to the eye — a common outcome of copying data out of a spreadsheet or a table, where cell padding sometimes carries over as literal whitespace. Trimming each line before comparison removes exactly this class of false non-duplicate, catching entries a human reviewer would immediately call the same but a strict character-for-character comparison would not.