Near-duplicate
A near-duplicate in eDiscovery is a document whose text content is highly similar to another's, typically above an agreed threshold such as 80 or 90 percent, but not identical, so exact-hash deduplication does not catch it. Near-duplicate analysis groups such documents around a pivot document so a reviewer can code the group consistently and see only the differences.
Also called: near-dupe · near dup · textual near-duplicate · similarity grouping
Updated September 6, 2026
How does near-duplicate analysis work?
After deduplication has removed exact copies, a large share of what remains is still repetitive: seven drafts of a contract, an email and the same email forwarded with a one-line comment, a scanned and an electronic version of the same letter. Near-duplicate analysis compares the extracted text of every document against every other, computes a similarity score, and clusters documents above the threshold around the most representative member, called the pivot.
In review, the group is presented together, differences are highlighted, and coding applied to the pivot can be propagated. The gain is consistency as much as speed: privilege or responsiveness decisions across a group of near-identical documents are made once, by one person.
What does near-duplicate analysis look like in practice?
| Document | Similarity to pivot | What differs |
|---|---|---|
| Supply_Agreement_v7.docx (pivot) | 100% | pivot document |
| Supply_Agreement_v6.docx | 96% | Section 4.2 pricing revised |
| Supply_Agreement_v7_signed.pdf | 94% | Signature blocks, scan artifacts |
| RE: Supply agreement (email) | 83% | Forward header, two-line comment |
How does near-duplicate analysis show up on an eDiscovery invoice?
Near-duplicate analysis is an analytics feature. On some platforms it is included in the license; on others it is an analytics surcharge per gigabyte or per matter per month, sometimes bundled with email threading and concept clustering. Its value is in review hours saved, which is usually the largest cost in the matter. See the volume translator for how review time scales with document count. Ask whether analytics are included or metered, and whether they can be switched on per matter.
Sources
- The Sedona Conference Glossary, “near-duplicate.”
- EDRM Analytics guidance.
Related terms
Deduplication · Email threading · Technology-assisted review · Culling
eDiscovery at one flat monthly rate
$3,250 a month on a 12-month agreement. No per-user fees and no per-GB hosting charges.
Common questions
What is the difference between a duplicate and a near-duplicate?
A duplicate is byte-for-byte identical and is identified by hash value. A near-duplicate shares most of its text but differs somewhere (a revised paragraph, a signature, a date) and is identified by comparing extracted text. Duplicates are removed; near-duplicates are grouped.
What similarity threshold is used?
Platforms let the user set it, with 100 percent meaning an identical text. Practitioners commonly work between 80 and 100 percent, and settings as high as 99 percent are used where the goal is culling rather than grouping. A lower threshold groups more aggressively and risks putting materially different documents together; a higher one leaves more singletons. The ESI protocol rarely fixes the number, since grouping affects review workflow rather than what is produced.
Are near-duplicates removed from production?
No. Each is a distinct document and is produced on its own merits. Near-duplicate grouping is a review efficiency: one reviewer codes the pivot and the group members in one pass, and inconsistent coding across the group is flagged.