deduplication repositories
Trending repositories tagged deduplication, ranked by trending score. Updated daily.
-
1
restic/restic
Fast, secure, efficient backup program
GitHub · Go · 37,323 stars · 1,988 forks · +29 stars in 24 hours · Trending score 3.06
-
2
windirstat/windirstat
Windows disk usage analyzer and cleanup tool with interactive treemaps, large-file discovery, duplicate detection, and file-system monitoring.
GitHub · C++ · 4,583 stars · 290 forks · +37 stars in 24 hours · Trending score 2.97
-
3
kopia/kopia
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
GitHub · Go · 14,757 stars · 811 forks · +13 stars in 24 hours · Trending score 2.68
-
4
CarlosZ16420/hamster-archiver
Windows local-first file organizer and media catalog with MCP tools for AI assistants. Verified batch archiving. 本地文件整理、媒体建库与可校验批量归档,支持 AI 通过 MCP 调用。
GitHub · JavaScript · 119 stars · 0 forks · +33 stars in 24 hours · Trending score 2.32
-
5
PlakarKorp/plakar
plakar is a backup solution powered by Kloset and ptar
GitHub · Go · 2,182 stars · 104 forks · +4 stars in 24 hours · Trending score 1.72
-
6
karanhudia/borg-ui
Replace complex Borg Backup terminal commands with a beautiful web UI. Create, schedule, and restore backups with just a few clicks.
GitHub · Python · 1,691 stars · 62 forks · +8 stars in 24 hours · Trending score 1.53
-
7
rustic-rs/rustic
rustic - fast, encrypted, and deduplicated backups powered by Rust
GitHub · Rust · 3,427 stars · 151 forks · 0 stars in 24 hours · Trending score 1.51
-
8
borgbackup/borg
Deduplicating archiver with compression and authenticated encryption.
GitHub · Python · 13,939 stars · 885 forks · +3 stars in 24 hours · Trending score 1.43
-
9
Ephemeral-AI-Lab/layerfs
Ephemeral Workspaces, durable shared history. LayerFS gives every agent a complete, writable filesystem Workspace over shared committed history, without copying the base. Agents work in parallel, commit what changed as deduplicated checkpoints, then branch, rewind or merge. Try it with LayerFS × DSH.
GitHub · Rust · 2,823 stars · 161 forks · 0 stars in 24 hours · Trending score 1.25
-
10
NVIDIA-NeMo/Curator
Scalable data pre processing and curation toolkit for LLMs
GitHub · Python · 1,866 stars · 379 forks · +1 stars in 24 hours · Trending score 1.04
-
11
sahib/rmlint
Extremely fast tool to remove duplicates and other lint from your filesystem
GitHub · C · 2,472 stars · 161 forks · 0 stars in 24 hours · Trending score 0.98
-
12
fcorbelli/zpaqfranz
Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
GitHub · C++ · 515 stars · 40 forks · +1 stars in 24 hours · Trending score 0.96
-
13
benseverndev-oss/goldenmatch
Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.
GitHub · Python · 139 stars · 18 forks · +1 stars in 24 hours · Trending score 0.83
-
14
eldertek/duplicatefinder
Discover and declutter! Find and eliminate duplicate files on Nextcloud.
GitHub · JavaScript · 59 stars · 15 forks · +1 stars in 24 hours · Trending score 0.73
-
15
NevermindNilas/TheAnimeScripter
Anime Video Toolkit: AI upscale, RIFE interpolation and restore in one pass. CLI, Standalone and After Effects.
GitHub · Python · 348 stars · 14 forks · 0 stars in 24 hours · Trending score 0.59
-
16
curlewlabs-com/apfs-dedupe
Safe APFS clone deduplication for macOS — fclones detection + a windowless, ACL-complete apply
GitHub · Shell · 8 stars · 1 forks · 0 stars in 24 hours · Trending score 0.47
-
17
maxkhim/laravel-storage-dedupler
Laravel package to prevent file duplications
GitHub · PHP · 58 stars · 3 forks · 0 stars in 24 hours · Trending score 0.39
-
18
kdeldycke/mail-deduplicate
📧 CLI to deduplicate mails from mail boxes
GitHub · Python · 203 stars · 47 forks · 0 stars in 24 hours · Trending score 0.37
-
19
rknightion/paperless-ngx-dedupe
Find and remove duplicate documents in paperless-ngx. MinHash/LSH fuzzy matching catches near-duplicates that exact hashing misses, with optional AI metadata suggestions.
GitHub · TypeScript · 28 stars · 2 forks · −1 stars in 24 hours · Trending score 0.33
-
20
FerrLabs/LFSX
A fast, lightweight, secure Git LFS server
GitHub · Rust · 7 stars · 1 forks · 0 stars in 24 hours · Trending score 0.32
-
21
STEXS-Technologies/shardline
Self-hosted, content-addressed storage infrastructure for AI models, datasets, containers, and build artifacts, with deduplication, integrity verification, and resilient data operations.
GitHub · Rust · 22 stars · 5 forks · 0 stars in 24 hours · Trending score 0.29
-
22
mhx/dwarfs
A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows
GitHub · C++ · 2,633 stars · 91 forks · 0 stars in 24 hours · Trending score 0.27
-
23
someplaceguy/hashsync
Run a command over SSH, deduplicating its output or input against files stored on the receiving host, to save bandwidth.
GitHub · Rust · 9 stars · 0 forks · 0 stars in 24 hours · Trending score 0.24
-
24
shootthesound/tensorvillage
One organized drive for every AI model you own - however many disks they are scattered across. Windows.
GitHub · 11 stars · 1 forks · 0 stars in 24 hours · Trending score 0.14