Deduplication
Deduplication in eDiscovery is the removal of exact duplicate documents from a collection before review, identified by comparing hash values, MD5 or SHA-1, computed from each file's content or, for email, from a combination of its metadata fields. Global deduplication removes copies across all custodians; custodial deduplication removes them only within each custodian's data.
Also called: dedupe · de-duplication · global deduplication · custodial deduplication · hash deduplication
Updated September 6, 2026
How does deduplication work?
Every collected file is run through a hashing algorithm that produces a fixed-length fingerprint, the hash value. Two files with identical content produce identical hashes; a one-byte difference produces a completely different one. The platform keeps the first instance of each hash and suppresses the rest.
For loose files that is enough. For email, the exported message file includes mailbox-specific data, so platforms hash a canonical set of message fields instead. Attachments are hashed separately and deduplicated as family members: an attachment is not suppressed if its parent email is unique.
Global deduplication is the norm because reviewing the same document once per custodian multiplies review cost with no gain. Its cost is that the produced set no longer shows, on its face, who held what, which is why the ALLCUSTODIANS field in the load file matters.
On content-addressed storage, deduplication is intrinsic rather than a processing step: identical content resolves to the same address, so a second copy is never stored. Storage of that kind makes deduplication a property of the system rather than a job that has to be run.
What does deduplication look like in practice?
Three custodians each hold the same 40-page board deck. Custodial deduplication: three copies processed, hosted and reviewed. Global deduplication: one copy reviewed, with ALLCUSTODIANS = Mendez; Apex; Lee in the load file. At 55 documents per reviewer-hour, the difference on one deck is trivial; across 300,000 documents with typical duplication rates it is weeks of review.
How many documents a gigabyte holds in the first place is covered in the pages-per-gigabyte guide.
How does deduplication show up on an eDiscovery invoice?
Deduplication reduces the volume that processing and hosting fees are charged on, but only if it happens before those meters run. Some providers process and bill on raw collected volume and deduplicate afterwards; others charge a separate deduplication or ECA fee per gigabyte for the reduction. Ask whether processing is billed pre- or post-deduplication, whether global deduplication is the default, and whether hosting is charged on the deduplicated set. The answer can change the monthly bill by a third or more.
Sources
- The Sedona Conference Glossary, “de-duplication.”
- EDRM Processing Standards on hashing and duplicate identification.
- NIST FIPS 180-4 (Secure Hash Standard) for SHA-1 and SHA-2.
Related terms
Hash value · Near-duplicate · Email threading · Custodian · Processing · Culling
eDiscovery at one flat monthly rate
$3,250 a month on a 12-month agreement. No per-user fees and no per-GB hosting charges.
Common questions
What is the difference between global and custodial deduplication?
Global removes every copy of a document after the first, across all custodians, and records the other holders in an ALLCUSTODIANS field. Custodial removes duplicates only within a single custodian's data, so a document held by five people is reviewed five times. Global is now the usual default; the ESI protocol should say which applies.
How is email deduplicated?
Not by file hash alone, because the same message exported from two mailboxes produces different files. Platforms compute a hash from a set of message fields (typically sender, recipients, subject, sent date and time, body and attachment hashes) so that the same message is recognized wherever it was collected. The field set varies by platform and should be stated in the protocol.
Does deduplication remove near-duplicates?
No. Deduplication removes only exact matches. Documents that differ by a signature block, a forwarded header or a single edit are near-duplicates, identified by a separate similarity analysis and normally grouped for review rather than removed.