fastdedupe

What it does

• Scans a folder tree and groups files by size, then verifies matches with a content hash (xxHash) — never by filename or extension.
• Reports before it touches anything. Run it with no flags and it only prints what it found; nothing is deleted or changed.
• Deletes or hardlinks copies on request, always keeping one file per duplicate set — asks for confirmation unless you pass --yes.
• Skips what it can't verify. A file it can't read is excluded and reported, never silently guessed about.
• Never goes silent. Duplicate groups print as soon as each one is found, not buffered until the whole scan finishes — with a progress line every few seconds, so a scan over a multi-terabyte archive still shows results the whole way through.

Using it

Just report, touch nothing:

fastdedupe ~/Downloads

See what would be deleted, without deleting it:

fastdedupe --delete --dry-run ~/Downloads

Delete copies, asking once for confirmation:

fastdedupe --delete ~/Downloads

Replace copies with hardlinks instead of deleting them — same content, one copy on disk, every path still works:

fastdedupe --link --yes ~/Downloads

How it decides what to keep

When several files share content, fastdedupe keeps the one whose folder holds the most other files — a stray copy sitting alone in a random folder is far more likely to be the extra one than a file sitting among nine siblings. Ties are broken by the older file. Two paths that are already hardlinks to the same file are treated as one file, not a copy of each other.

What you need

The macOS and Windows binaries aren't code-signed. macOS will warn about an unidentified developer the first time — right-click the file, choose Open, and confirm once. Windows SmartScreen behaves the same way — click More info → Run anyway.