Remove Duplicates From CSV — By Key Columns, Locally

Remove duplicate rows from a CSV

Merged exports, re-run scrapes, doubled form submissions — duplicates creep into CSVs constantly, and Excel’s Remove Duplicates means opening the file in Excel (with everything that does to your data) and being under its million-row ceiling. Here, dedupe is one streaming pass in your browser: drop the file, choose what “duplicate” means, and watch the removed count appear — 7 rows or 70,000, it works the same on a 2 GB file.

The important subtlety is that “duplicate” is a decision, not a fact. Whole rows identical? Same email regardless of case? Same order ID even when the rest differs? This tool makes the decision explicit: dedupe on all columns or on the key columns you pick, with case- and whitespace-insensitivity on by default (because “ Ada ” and “ada” are almost always the same person wearing different formatting).

Preset: the Clean panel opens on load with duplicate removal in front.

Drop a CSV, TSV or Excel file here

.csv · .tsv · .txt · .csv.gz · .xlsx — files up to 4 GB open here, in your browser. Nothing is uploaded; the file is read in place on your device.

This viewer needs JavaScript — that’s what lets it read the file on your device instead of uploading it to a server. Enable JavaScript for this page and reload.

How it works

Drop the file. Open — the Clean panel is already up.
Pick key columns (or keep “All columns” for exact-row duplicates) and the case/whitespace options.
Hit Remove duplicates — the removed count shows immediately; the first occurrence of each key is kept.
Export the deduplicated file. “Undo all cleanups” restores everything if you chose the wrong key.

Choosing the right duplicate key

All columns is the safe default: only rows identical in every field are collapsed, so it can’t merge two genuinely different records. It also misses near-duplicates — the same customer with a re-typed phone number survives. Key columns (say, email alone) catch those, at the cost of a judgment call: the first row with each email is kept and later ones dropped wholesale, including any fields where the dropped row was actually fresher. For contact lists, email or email+name is the usual key; for transactions, the order/transaction ID; for address data, deduping on address+ZIP catches household duplicates.

Sort order matters because first occurrence wins. If you want the newest record kept, sort by your date column descending first, then dedupe — the streaming pass sees rows in the sorted order. That two-step (sort, then dedupe) is the whole “keep latest per key” recipe.

Good to know

Preview what you’re about to lose: filter to a suspect key first (search an email), eyeball the copies, then dedupe.
Trim-insensitivity is on by default because trailing spaces are the #1 cause of “why wasn’t this caught as a duplicate”.
The removed rows aren’t gone until you export — “Undo all cleanups” brings them back instantly.

Frequently asked questions

Which of the duplicate rows is kept?

The first one the file contains (in the current sort order). Sort before deduping to control which copy that is — e.g. sort by updated_at descending to keep the most recent record per key.

Can I deduplicate on one column only, like email?

Yes — untick “All columns” in the Clean panel and tick just the key column(s). Rows sharing that key are collapsed to the first occurrence, regardless of differences elsewhere.

Does it treat “ADA@EXAMPLE.COM” and “ada@example.com ” as duplicates?

With the defaults, yes — comparison ignores case and surrounding whitespace, which matches how humans read the data. Both options can be switched off for byte-exact comparison.

How big a file can be deduplicated?

The same limits as the rest of the site (files to 4 GB). One honest caveat: dedupe keeps a fingerprint of every unique key in memory, so tens of millions of long unique keys is where a laptop may feel it. Typical files — even millions of rows — are a few seconds.