Processing

How does processing work?

Collected data is not documents. A mailbox is one large PST file; a shared drive is a tree of folders holding archives inside archives; a phone extraction is a database. Processing is the pipeline that turns that into a set of discrete, identified, searchable documents with a stable identifier each.

The first pass is expansion and identification. Containers are opened recursively, and each file is identified by its binary signature rather than its extension, so a spreadsheet renamed .txt is still treated as a spreadsheet. Family relationships are recorded (which attachment belongs to which parent) because they must survive to production.

The second pass is reduction and extraction. Each file is hashed; de-NISTing removes known software components and deduplication removes exact copies. Text is extracted from the electronic original where it exists and generated by OCR where it does not. Metadata is pulled from the file system and from inside the file, and dates are normalized to one stated time zone, usually UTC, so that a sort by date means something across custodians in different offices.

The third pass is indexing. The extracted text and the metadata fields are written to a search index, which is what makes term searching, threading and analytics possible at all.

What does processing look like in practice?

A 40 GB mailbox collection is roughly 400,000 email documents at the 10,000-documents-per-gigabyte rule of thumb the volume translator uses, drawn from published analysis of about 2,000 hosted matters. Processing expands the PSTs into those documents, hashes them, drops the exact duplicates, extracts sender, recipients, subject and sent date into fields, and indexes the bodies and attachments. What arrives in review is a searchable document set; what arrives on the exception report might be 300 encrypted attachments nobody can read yet.

(Document counts per gigabyte are rules of thumb and vary with attachment density.)

How does processing show up on an eDiscovery invoice?

Per-GB processing is the classic one-time volume fee, and the Fee Field Guide sets out how it is billed. Watch for the same gigabyte being charged more than once on its way in: a separate ingestion or loading fee and a vaguely named data enrichment fee for normalization, metadata extraction and de-NISTing are charges for steps that are already part of any modern pipeline.

OCR is often broken out per page rather than included. So is ECA and culling, which means paying for the reduction and then paying to host what survives it.

Processing is one of the six meters on a per-GB invoice; the cost guide lists them all and the per-GB guide works out what a single gigabyte costs across a matter’s life.

Worth asking: is processing charged on pre-cull or post-cull volume; is loading included; what is the combined per-GB cost from raw data to reviewable document; and is re-processing charged again if data is reloaded after a fix.

Sources

• EDRM, Processing Standards.
• The Sedona Conference, Glossary: eDiscovery & Digital Information Management, “processing.”
• Documents-per-gigabyte figures: src/data/volume.json, from published analysis of ~2,000 hosted matters and 150M+ documents.
• Fee ranges and documented rates: the Fee Field Guide dataset (ComplexDiscovery pricing survey; filed court cost schedules).
• RAND Institute for Civil Justice, Where the Money Goes (2012): processing about 19 percent of production cost, against 73 percent for review.