Quality Control and Assurance in Document Review
Expert-defined terms from the Advanced Certification in Legal Document Review (United Kingdom) course at London School of Business and Administration. Free to read, free to share, paired with a professional course.
Audit Trail – a chronological record of all actions taken on a document d… #
Related terms: Log file, provenance, metadata. Example: A reviewer marks a clause, the system logs the time, reviewer ID, and change type. Practical application: Enables supervisors to trace decisions and verify compliance with review protocols. Challenges: Ensuring the trail is tamper‑proof and that it captures sufficient detail without overwhelming storage resources.
Batch Review – the process of grouping a set of documents for simultaneou… #
Related terms: Work packet, docket, throughput. Example: A team receives 500 emails to be screened for privileged information; they are divided into batches of 50. Practical application: Facilitates parallel processing and easier monitoring of progress. Challenges: Maintaining uniform quality across batches and preventing fatigue‑related errors.
Bias Mitigation – strategies employed to reduce personal or systemic bias… #
Related terms: Cognitive bias, blind review, calibration. Example: Reviewers rotate assignments so no single reviewer consistently handles sensitive topics. Practical application: Enhances objectivity and defensibility of findings. Challenges: Detecting subtle biases and allocating sufficient training time.
Calibration Session – a structured meeting where reviewers discuss sample… #
Related terms: Consensus building, standardisation, training. Example: A senior attorney explains the criteria for “confidential” versus “public” classification. Practical application: Improves inter‑reviewer reliability. Challenges: Scheduling, ensuring all participants engage fully, and updating criteria as case law evolves.
Chain of Custody – documented sequence of handling and storage of electro… #
Related terms: Evidence preservation, audit trail, provenance. Example: A PDF is transferred from the client’s server to the review platform, with logs confirming each hand‑off. Practical application: Critical for litigation where document authenticity may be contested. Challenges: Managing multiple transfers without breaking the chain, especially across jurisdictions.
Check‑list Compliance – use of predefined lists to verify that each revie… #
Related terms: Quality gate, standard operating procedure, verification. Example: Before finalising a batch, reviewers confirm that all privileged tags have been applied. Practical application: Provides a tangible control point. Challenges: Checklist fatigue and the temptation to “tick‑box” without genuine verification.
Codebook – a reference guide detailing the coding schema, definitions, an… #
Related terms: Taxonomy, annotation guide, metadata schema. Example: The codebook defines “relevant” as any document containing at least one claim related to the breach of contract. Practical application: Ensures consistent tagging across the review team. Challenges: Keeping the codebook up to date as new issues emerge and ensuring accessibility for all reviewers.
Compliance Audit – an independent assessment to verify that document revi… #
Related terms: Quality assurance, external audit, risk assessment. Example: An external firm reviews a law firm’s handling of GDPR‑sensitive data during discovery. Practical application: Identifies gaps and drives corrective action. Challenges: Balancing thoroughness with the time‑sensitive nature of litigation.
Confidentiality Protocol – procedures that protect privileged or sensitiv… #
Related terms: Data security, access control, privilege log. Example: Reviewers must use a secure VPN and cannot copy documents to personal devices. Practical application: Safeguards client‑attorney privilege. Challenges: Enforcing protocols across remote teams and integrating them with existing IT infrastructure.
Control Sample – a subset of reviewed documents selected for detailed qua… #
Related terms: Random sampling, statistical quality control, audit. Example: 5% Of a batch is re‑examined by a senior attorney for consistency. Practical application: Provides measurable insight into error rates. Challenges: Choosing a representative sample size and avoiding selection bias.
Cross‑Check Review – a method where two reviewers independently assess th… #
Related terms: Double coding, peer review, reconciliation. Example: Reviewer A marks a paragraph as “relevant”; Reviewer B marks it as “non‑relevant”, prompting a discussion. Practical application: Enhances reliability and uncovers oversights. Challenges: Increased workload and the need for clear escalation procedures.
Data Integrity – the accuracy, completeness, and consistency of document… #
Related terms: Data validation, corruption, checksum. Example: A checksum is generated for each file to detect any alteration during transfer. Practical application: Assures courts that evidence has not been tampered with. Challenges: Managing large volumes of data and preventing inadvertent changes during redaction.
Document Custodian – the individual or team responsible for maintaining a… #
Related terms: Custodian interview, data steward, repository manager. Example: The IT department acts as custodian for server logs. Practical application: Centralises responsibility and streamlines communication with counsel. Challenges: Aligning custodian availability with tight discovery deadlines.
Document Management System (DMS) – software platform used to store, organ… #
Related terms: Repository, e‑discovery platform, version control. Example: Relativity or Concordance is used to host the case file set. Practical application: Provides searchable access and integrates with QA tools. Challenges: Ensuring the DMS supports audit trails and can handle large data sets without performance degradation.
Document Redaction – the process of obscuring or removing sensitive infor… #
Related terms: Sanitisation, blurring, privileged material. Example: A social security number is blacked out in a PDF before production. Practical application: Protects privacy and complies with data protection laws. Challenges: Avoiding over‑redaction that removes relevant content and ensuring redactions are irreversible.
Document Review Workflow – the sequence of steps a document follows from… #
Related terms: Process map, pipeline, lifecycle. Example: Intake → allocation → coding → QA → export. Practical application: Visualising the workflow helps identify bottlenecks. Challenges: Adapting the workflow to case‑specific nuances without compromising standard controls.
Electronic Discovery Reference Model (EDRM) – a framework that outlines t… #
Related terms: Discovery lifecycle, data mapping, preservation. Example: The “preservation” phase requires issuing legal hold notices to relevant custodians. Practical application: Guides teams in applying appropriate QA measures at each stage. Challenges: Aligning the model with the firm’s internal processes and technology stack.
Error Rate – the proportion of documents incorrectly coded or missed duri… #
Related terms: Defect density, quality metric, false positive. Example: An audit finds a 2% error rate in privilege tagging. Practical application: Benchmarks performance and informs corrective actions. Challenges: Calculating error rates accurately when ground truth is unavailable.
Escalation Protocol – a predefined set of actions for handling disagreeme… #
Related terms: Issue resolution, tiered review, supervision. Example: If a reviewer flags a document as “potentially privileged”, it is escalated to a senior attorney for final determination. Practical application: Ensures consistent handling of high‑risk items. Challenges: Preventing bottlenecks when escalations accumulate.
External Quality Assurance (EQA) – an independent assessment performed by… #
Related terms: Peer audit, external audit, certification. Example: A professional services firm conducts an EQA to confirm compliance with ISO 17025 standards. Practical application: Provides credibility to clients and regulators. Challenges: Coordinating schedules and access while maintaining confidentiality.
Feedback Loop – a continuous process where reviewers receive performance… #
Related terms: Corrective action, coaching, performance metrics. Example: After a QA audit, reviewers receive a report highlighting common coding errors. Practical application: Drives learning and reduces repeat mistakes. Challenges: Delivering feedback constructively and ensuring it leads to measurable change.
File Integrity Check – verification that a document’s content remains unc… #
Related terms: Checksum, MD5, SHA‑256. Example: Before uploading files to the review platform, a SHA‑256 hash is generated and later compared post‑upload. Practical application: Assures that evidence has not been altered. Challenges: Managing hash calculations for large volumes and handling false positives due to legitimate format conversions.
Granular Tagging – the practice of applying specific, detailed tags to in… #
Related terms: Annotation, micro‑coding, segment tagging. Example: A single contract is tagged at clause level for “indemnity”, “termination”, and “confidentiality”. Practical application: Improves searchability and precision in production. Challenges: Increased time required and the need for clear tagging guidelines.
Ground‑Truth Dataset – a set of documents whose correct classifications a… #
Related terms: Reference set, gold standard, validation set. Example: A curated collection of 200 privileged emails is used to assess reviewer accuracy. Practical application: Provides an objective measure of performance. Challenges: Creating a sufficiently representative ground‑truth without bias.
Heuristic Review – applying rule‑of‑thumb techniques to quickly identify… #
Related terms: Keyword search, predictive coding, triage. Example: Documents containing the phrase “attorney‑client” are flagged for privileged review. Practical application: Speeds up early phases of large‑scale projects. Challenges: Risk of overlooking nuanced privilege claims that do not contain obvious keywords.
Impact Assessment – evaluation of the potential consequences of a quality… #
Related terms: Risk analysis, severity rating, mitigation plan. Example: A missed privileged document could expose the client to sanctions; the impact assessment rates this as high. Practical application: Prioritises remediation efforts. Challenges: Quantifying intangible effects such as reputational damage.
In‑House QA Team – a dedicated group within a law firm or corporation res… #
Related terms: Quality manager, compliance officer, internal audit. Example: The QA team conducts weekly spot‑checks of coding accuracy. Practical application: Embeds quality expertise directly into the review workflow. Challenges: Resource allocation and maintaining independence from the review staff.
Information Governance – policies and procedures that manage the creation… #
Related terms: Data lifecycle, records management, policy framework. Example: A firm’s information governance policy dictates that all discovery data be retained for seven years. Practical application: Provides a foundation for QA controls. Challenges: Aligning governance with fast‑moving litigation demands.
Keyword Search – the use of specific words or phrases to locate potential… #
Related terms: Boolean query, search string, query builder. Example: The search term “breach of warranty” returns 1,200 documents for review. Practical application: Serves as an initial filter before manual coding. Challenges: Over‑reliance can miss concept‑based relevance and generate false positives.
Legal Hold – an instruction to preserve all relevant data that may be sub… #
Related terms: Preservation notice, hold notice, data retention. Example: After a claim is filed, the IT department receives a legal hold notice to suspend routine backups. Practical application: Safeguards evidence integrity. Challenges: Ensuring all custodians understand and comply with the hold across multiple jurisdictions.
Metadata Management – the practice of controlling the data that describes… #
Related terms: Data attributes, schema, tagging. Example: Each PDF is assigned metadata fields for “client”, “matter”, and “review status”. Practical application: Enhances traceability and facilitates reporting. Challenges: Inconsistent metadata entry and the need for automated extraction tools.
Model Validation – the process of confirming that predictive coding or ma… #
Related terms: Training set, test set, precision, recall. Example: After training a model on 5,000 documents, a validation set shows 92% recall. Practical application: Justifies reliance on technology‑assisted review. Challenges: Selecting unbiased validation samples and interpreting statistical metrics for legal stakeholders.
Peer Review – a quality control activity where a reviewer’s work is exami… #
Related terms: Double‑check, cross‑check, supervision. Example: A junior reviewer’s coding is reviewed by a senior associate before finalisation. Practical application: Catches errors early and provides learning opportunities. Challenges: Allocating sufficient time without delaying project timelines.
Performance Dashboard – visual reporting tool displaying key quality metr… #
Related terms: KPI, analytics, scorecard. Example: A dashboard shows a 1.5% Error rate and 120 documents per hour for each reviewer. Practical application: Enables real‑time monitoring and rapid response to issues. Challenges: Ensuring data accuracy and avoiding metric overload.
Predictive Coding – the use of machine‑learning algorithms to automatical… #
Related terms: Technology‑assisted review, TAR, supervised learning. Example: After training, the system predicts 80% of the remaining documents as non‑relevant, reducing manual effort. Practical application: Dramatically cuts review costs for large datasets. Challenges: Gaining client acceptance, ensuring transparency, and maintaining defensibility in court.
Privilege Log – a detailed record of all documents claimed as privileged,… #
Related terms: Confidential log, privilege register, metadata. Example: The log lists a letter dated 12 March 2023, “Attorney‑Client Communication”, without attaching the letter itself. Practical application: Satisfies disclosure obligations while protecting privilege. Challenges: Balancing sufficient description with the risk of waiving privilege.
Quality Assurance (QA) – systematic activities designed to ensure that do… #
Related terms: Quality control, assurance framework, audit. Example: QA includes random sampling, reviewer calibration, and error‑rate monitoring. Practical application: Builds confidence in the review results for clients and courts. Challenges: Integrating QA without creating excessive overhead and maintaining consistency across dispersed teams.
Quality Control (QC) – operational controls applied during the review pro… #
Related terms: Inspection, verification, corrective action. Example: A QC check flags a document where the reviewer missed a privilege tag, prompting immediate correction. Practical application: Prevents defects from propagating to production. Challenges: Ensuring QC checks are timely and not perceived as punitive.
Random Sampling – selecting documents without bias to assess overall revi… #
Related terms: Statistical sampling, control sample, audit. Example: 3% Of each batch is randomly chosen for senior review. Practical application: Provides a statistically valid measure of error rates. Challenges: Determining an appropriate sample size to achieve confidence levels while managing workload.
Redaction Review – a specialized QA step that verifies that all redaction… #
Related terms: Sanitisation check, blind spot analysis, compliance audit. Example: After production, a reviewer uses a tool to scan PDFs for residual text behind black boxes. Practical application: Prevents inadvertent disclosure. Challenges: Technical limitations of some redaction software and the time required for thorough checks.
Relevancy Scoring – assigning a numeric value to documents based on their… #
Related terms: Ranking, confidence level, threshold. Example: Documents scoring above 0.8 Are automatically marked “relevant”. Practical application: Prioritises reviewer attention on high‑value material. Challenges: Setting appropriate thresholds and explaining scoring methodology to non‑technical stakeholders.
Review Protocol – a documented set of procedures that outlines how review… #
Related terms: SOP, standard operating procedure, workflow guide. Example: The protocol mandates that all emails be screened for attachments before coding. Practical application: Standardises actions across the team. Challenges: Keeping the protocol current with evolving case law and technology.
Risk‑Based QA – allocating quality resources proportionally to the percei… #
Related terms: Risk assessment, prioritisation, mitigation. Example: Privileged documents receive a higher QC frequency than generic correspondence. Practical application: Optimises effort where it matters most. Challenges: Accurately assessing risk levels and avoiding blind spots.
Sample Size Determination – statistical calculation used to decide how ma… #
Related terms: Confidence interval, margin of error, power analysis. Example: For a 95% confidence level with a 2% margin of error, 2,400 documents are required from a pool of 100,000. Practical application: Ensures QA findings are statistically valid. Challenges: Balancing statistical rigor with operational constraints.
Search Term Optimization – refining keyword queries to improve recall and… #
Related terms: Query refinement, Boolean logic, relevance feedback. Example: Adding “non‑disclosure” to the original search string increased relevant hits by 12%. Practical application: Reduces unnecessary manual review. Challenges: Avoiding overly complex queries that miss subtle concepts.
Segmented Review – dividing a document into logical parts (e #
G., Clauses, paragraphs) and reviewing each segment separately. Related terms: Granular tagging, micro‑coding, modular analysis. Example: A 50‑page contract is broken into 250 clauses for individual assessment. Practical application: Enhances precision and facilitates targeted production. Challenges: Increased time for segmentation and potential for inconsistent segment boundaries.
Standard Operating Procedure (SOP) – a detailed, step‑by‑step instruction… #
Related terms: Protocol, workflow guide, best practice. Example: The SOP for “privilege verification” outlines the exact steps for checking redactions and logging decisions. Practical application: Provides clear guidance and reduces variation. Challenges: Ensuring SOPs are user‑friendly and updated promptly.
Statistical Quality Control (SQC) – the application of statistical method… #
Related terms: Control chart, process capability, variance analysis. Example: A control chart shows error rates staying within the predefined upper control limit. Practical application: Enables early detection of process drift. Challenges: Interpreting statistical signals correctly and training staff in SQC concepts.
Tagging Consistency – the degree to which reviewers apply the same codes… #
Related terms: Coding uniformity, agreement index, kappa statistic. Example: A kappa score of 0.85 Indicates high tagging consistency across the team. Practical application: Supports defensibility of the review outcome. Challenges: Achieving high consistency in complex or ambiguous document sets.
Technical Review – an assessment focused on the functionality and perform… #
Related terms: System audit, IT validation, performance testing. Example: A technical review confirms that the platform encrypts data at rest and in transit. Practical application: Ensures the technology does not compromise QA objectives. Challenges: Coordinating between legal and IT teams and keeping up with software updates.
Test Set Evaluation – assessing the performance of a predictive coding mo… #
Related terms: Validation set, hold‑out set, model testing. Example: The test set yields a precision of 94% and recall of 89%. Practical application: Provides confidence that the model will perform on the full dataset. Challenges: Selecting a test set that truly reflects the diversity of the case.
Thoroughness Ratio – a metric comparing the number of documents reviewed… #
Related terms: Completeness, coverage metric, audit percentage. Example: A 98% thoroughness ratio shows that almost all documents have been examined. Practical application: Demonstrates diligence to the client. Challenges: Balancing thoroughness with cost and time constraints.
Time‑Stamped Review – recording the exact time each action (open, tag, ap… #
Related terms: Audit trail, logging, chronology. Example: The system logs that Reviewer 3 coded Document A at 09:23 GMT on 5 May 2026. Practical application: Vital for demonstrating compliance with court‑ordered deadlines. Challenges: Ensuring timestamps are synchronized across distributed servers.
Training Matrix – a structured overview showing which reviewers have comp… #
Related terms: Skill matrix, competency chart, learning log. Example: The matrix indicates that all senior reviewers have completed “privilege identification” training. Practical application: Helps managers allocate work appropriately. Challenges: Keeping the matrix current as staff turnover occurs.
Validation Protocol – a set of procedures for confirming that QA tools, s… #
Related terms: Verification, acceptance testing, quality gate. Example: The protocol requires that a redaction tool be tested on 100 sample PDFs with known sensitive data. Practical application: Prevents tool‑related errors from entering the review. Challenges: Allocating resources for thorough validation in fast‑track projects.
Version Control – the management of document revisions to ensure that rev… #
Related terms: Revision history, check‑in/check‑out, baseline. Example: Document B is locked when Reviewer 1 opens it, preventing another reviewer from editing simultaneously. Practical application: Eliminates duplication and conflicting edits. Challenges: Handling large numbers of versions and ensuring proper archiving.
Workflow Automation – the use of software rules to automatically route do… #
Related terms: Rule‑engine, robotic process automation, orchestration. Example: When a document receives a “privileged” tag, the system automatically sends it to the senior attorney for final sign‑off. Practical application: Reduces manual coordination and speeds up the process. Challenges: Designing robust rules that accommodate exceptions without causing bottlenecks.
Yield Rate – the proportion of reviewed documents that are deemed relevan… #
Related terms: Productivity, relevance ratio, extraction efficiency. Example: A yield rate of 7% means that 7 out of every 100 documents reviewed contain material pertinent to the claim. Practical application: Assists in budgeting and resource planning. Challenges: Low yield can demotivate reviewers and increase cost per useful document.
Zero‑Error Policy – an aspirational stance that seeks to eliminate all mi… #
Related terms: Perfectionism, continuous improvement, error‑free objective. Example: The policy mandates immediate re‑review of any document flagged by QA. Practical application: Underscores the importance of meticulous work. Challenges: Realistic attainment is difficult; the policy can create pressure and may lead to diminishing returns if not balanced with pragmatic risk assessments.