Statistical Sampling and Validation Methods
Expert-defined terms from the Advanced Certification in Legal Document Review course at London School of Business and Administration. Free to read, free to share, paired with a globally recognised certification pathway.
Statistical Sampling and Validation Methods #
Statistical Sampling and Validation Methods
Statistical Sampling and Validation Methods are crucial techniques used in the f… #
This glossary will cover key terms and concepts related to statistical sampling and validation methods in the context of the Advanced Certification in Legal Document Review.
1 #
Statistical Sampling
Statistical Sampling is the process of selecting a representative subset of data… #
In legal document review, statistical sampling is used to analyze a sample of documents rather than reviewing every single document in a dataset. This method helps save time and resources while still providing a reliable estimate of the characteristics of the entire document collection.
Example #
A legal team may use statistical sampling to review a random sample of emails in a large dataset to identify relevant evidence for a case.
2 #
Validation Methods
Validation Methods are techniques used to assess the accuracy and reliability of… #
Validation methods help ensure that the sample selected is truly representative of the population and that the conclusions drawn from the sample are valid.
Example #
After conducting a statistical sample review of a document collection, validation methods may include comparing the results of the sample review with a manual review of the entire dataset to ensure consistency.
3 #
Confidence Level
The Confidence Level is a statistical measure that indicates the degree of certa… #
It is usually expressed as a percentage, representing the likelihood that the true value lies within a certain range.
Example #
A confidence level of 95% means that there is a 95% probability that the results obtained from a statistical sample are accurate and representative of the entire population.
4 #
Margin of Error
The Margin of Error is a measure of the uncertainty or variability in the result… #
It indicates the range within which the true value of a parameter is likely to lie.
Example #
If a statistical sample review produces a result with a margin of error of ±3%, it means that the true value is likely to be within 3 percentage points of the reported result.
5 #
Sampling Frame
A Sampling Frame is a list or database that contains the elements of the populat… #
It serves as the basis for selecting the sample and ensures that all elements of the population have an equal chance of being included in the sample.
Example #
In a legal document review, a sampling frame may consist of a list of all documents in a dataset that meet certain criteria, such as relevance to a specific case.
6 #
Population
The Population refers to the entire set of elements or units that are of interes… #
In legal document review, the population may consist of all documents in a dataset that need to be reviewed for a specific purpose.
Example #
The population for a legal document review project may include all emails, contracts, and other documents related to a particular legal case.
7 #
Sample Size
The Sample Size is the number of elements or units selected from the population… #
The size of the sample is crucial in determining the accuracy and reliability of the results obtained from the sample.
Example #
A legal team may decide to review a sample of 500 emails from a dataset of 10,000 emails to assess the relevance of the documents to a legal case.
8 #
Random Sampling
Random Sampling is a sampling technique in which every element in the population… #
This method helps eliminate bias and ensures that the sample is representative of the population.
Example #
In a legal document review, random sampling may involve selecting documents from a dataset using a random number generator to ensure that every document has an equal chance of being included in the sample.
9 #
Stratified Sampling
Stratified Sampling is a sampling technique that involves dividing the populatio… #
This method helps ensure that all subgroups are adequately represented in the sample.
Example #
In a legal document review, documents may be stratified based on their type (e.g., emails, contracts, memos) before selecting a sample from each stratum for review.
10 #
Cluster Sampling
Cluster Sampling is a sampling technique in which the population is divided into… #
This method is useful when it is impractical to sample individual elements from the population.
Example #
In a large document collection, cluster sampling may involve selecting a random sample of folders or directories containing documents for review rather than selecting individual documents.
11 #
Systematic Sampling
Systematic Sampling is a sampling technique in which every kth element in the po… #
This method is simple and efficient but may introduce bias if there is a pattern in the data.
Example #
In a legal document review, systematic sampling may involve selecting every 10th document from a dataset of emails for review to create a sample.
12 #
Judgmental Sampling
Judgmental Sampling is a non #
probabilistic sampling technique in which the sample is selected based on the judgment and expertise of the reviewer rather than through random selection. This method is subjective and may introduce bias but can be useful when specific expertise is required.
Example #
In a legal document review, judgmental sampling may involve selecting documents that are known to be relevant to a case based on the judgment of experienced legal reviewers.
13 #
Convenience Sampling
Convenience Sampling is a non #
probabilistic sampling technique in which the sample is selected based on convenience or accessibility. This method is quick and easy but may not be representative of the population.
Example #
In a legal document review, convenience sampling may involve selecting documents that are readily available or easily accessible for review without considering their representativeness.
14. Non #
Sampling Error
Non #
Sampling Error refers to errors that occur in a statistical study that are not related to the sampling process itself. These errors can arise from various sources, such as data collection methods, measurement errors, or data processing.
Example #
Non-sampling errors in a legal document review may include inaccuracies in document metadata, misinterpretation of legal terms, or inconsistencies in reviewer judgments.
15 #
Sampling Error
Sampling Error refers to the difference between the results obtained from a stat… #
This error arises due to the variability inherent in sampling and can be reduced by increasing the sample size.
Example #
If a statistical sample review of a document collection produces a result with a margin of error of ±5%, the sampling error is 5 percentage points.
16. Cross #
Validation
Cross #
Validation is a validation technique used to assess the performance and generalizability of a predictive model by testing it on multiple subsets of the data. This method helps ensure that the model is robust and reliable across different datasets.
Example #
In a legal document review, cross-validation may involve testing the accuracy of a predictive coding model on multiple samples of documents from the same dataset to evaluate its consistency.
17. Inter #
Rater Reliability
Inter #
Rater Reliability is a measure of the consistency and agreement between different reviewers or raters in their assessments or judgments. This measure is important in legal document review to ensure that multiple reviewers are interpreting and coding documents consistently.
Example #
In a legal document review project, inter-rater reliability may be assessed by comparing the coding decisions of two or more reviewers on a sample of documents to determine the level of agreement.
18 #
Relevance Sampling
Relevance Sampling is a sampling technique that focuses on selecting documents o… #
This method helps prioritize the review of documents that are likely to contain important information.
Example #
In a legal document review, relevance sampling may involve selecting documents that contain specific keywords or phrases relevant to a legal case for further review.
19 #
Stratification Error
Stratification Error occurs when the subgroups or strata created for stratified… #
This error can affect the accuracy and reliability of the sample results.
Example #
If documents in a legal document review are stratified based on their type but there is significant variability within each type, the stratification error may lead to biased results.
20 #
Validation Set
A Validation Set is a separate subset of data that is used to assess the perform… #
This set is not used in the training of the model and serves as an independent test of the model's predictive power.
Example #
In a legal document review, a validation set may consist of a sample of documents that are held out from the training set used to develop a predictive coding model and are used to evaluate the model's performance.
21 #
Calibration Set
A Calibration Set is a subset of data used to adjust the parameters of a predict… #
This set is used to fine-tune the model and ensure that it is well-calibrated and accurate.
Example #
In a legal document review, a calibration set may be used to adjust the threshold for predicting document relevance in a predictive coding model based on the feedback from reviewers.
22 #
Holdout Set
A Holdout Set is a subset of data that is reserved for testing the performance o… #
This set helps evaluate the generalizability of the model to new data.
Example #
In a legal document review, a holdout set may consist of a sample of documents that are not used in the training or validation of a predictive coding model but are reserved for final testing.
23 #
Training Set
A Training Set is a subset of data used to build and train a predictive model or… #
This set is used to teach the model to recognize patterns and make predictions based on the input data.
Example #
In a legal document review, a training set may consist of a sample of documents that are manually reviewed and coded to train a predictive coding model to classify documents as relevant or non-relevant.
24 #
Overfitting
Overfitting is a phenomenon in which a predictive model or algorithm performs we… #
Overfitting occurs when the model is too complex or too closely fits the training data.
Example #
In a legal document review, overfitting may occur when a predictive coding model memorizes specific patterns in the training set that are not representative of the entire document collection, leading to poor performance on new documents.
25 #
Underfitting
Underfitting is the opposite of overfitting and occurs when a predictive model o… #
Underfitting results in poor performance on both the training and test data.
Example #
In a legal document review, underfitting may occur when a predictive coding model is too basic and fails to capture the nuances and complexities of the document collection, leading to inaccurate predictions.
26 #
Receiver Operating Characteristic (ROC) Curve
A Receiver Operating Characteristic (ROC) Curve is a graphical representation of… #
The curve plots the true positive rate against the false positive rate to evaluate the model's accuracy.
Example #
In a legal document review, an ROC curve may be used to assess the performance of a predictive coding model in distinguishing between relevant and non-relevant documents at different decision thresholds.
27 #
Area Under the Curve (AUC)
The Area Under the Curve (AUC) is a summary measure of the overall performance o… #
A higher AUC value indicates better discrimination between the positive and negative classes.
Example #
In a legal document review, a predictive coding model with an AUC of 0.85 performs better at distinguishing between relevant and non-relevant documents than a model with an AUC of 0.70.
28 #
Precision and Recall
Precision and Recall are two performance metrics used to evaluate the effectiven… #
Precision measures the proportion of true positive predictions among all positive predictions, while recall measures the proportion of true positives predicted correctly among all actual positive cases.
Example #
In a legal document review, precision and recall are used to assess how accurately a predictive coding model identifies relevant documents and avoids false positives.
29 #
F1 Score
The F1 Score is a single metric that combines both precision and recall into a s… #
The F1 score is calculated as the harmonic mean of precision and recall.
Example #
In a legal document review, the F1 score of a predictive coding model reflects the balance between its ability to correctly identify relevant documents (precision) and its ability to capture all relevant documents (recall).
30 #
Confusion Matrix
A Confusion Matrix is a table that summarizes the performance of a classificatio… #
The matrix helps visualize the model's performance and identify areas of improvement.
Example #
In a legal document review, a confusion matrix may be used to evaluate the predictive coding model's performance in classifying documents as relevant or non-relevant based on the actual and predicted labels.
31 #
Bootstrap Sampling
Bootstrap Sampling is a resampling technique in which multiple samples are drawn… #
This method helps assess the robustness of the results obtained from a single sample.
Example #
In a legal document review, bootstrap sampling may be used to generate multiple samples of documents from a dataset to estimate the confidence intervals around key metrics, such as precision and recall.
32 #
Monte Carlo Simulation
Monte Carlo Simulation is a computational technique that uses random sampling to… #
This method generates multiple simulations based on random inputs to estimate the range of possible outcomes.
Example #
In a legal document review, Monte Carlo simulation may be used to assess the impact of sampling variability on the results of a statistical analysis and to determine the level of uncertainty in the findings.
33. Cross #
Validation Error
Cross #
Validation Error is an estimate of the generalization error of a predictive model obtained through cross-validation. This error measures how well the model is likely to perform on new, unseen data based on its performance on the validation sets.
Example #
In a legal document review, cross-validation error may be used to assess the predictive coding model's ability to generalize to new documents and cases based on its performance on multiple validation sets.
34 #
Outlier Detection
Outlier Detection is a statistical technique used to identify data points that d… #
Outliers may indicate errors in the data, anomalies, or important patterns that require further investigation.
Example #
In a legal document review, outlier detection may be used to identify documents that contain unusual or suspicious content that may be relevant to a case or require special attention.
35 #
Data Imputation
Data Imputation is a method used to fill in missing values in a dataset by estim… #
Imputing missing values helps maintain the integrity and completeness of the dataset for analysis.
Example #
In a legal document review, data imputation may be used to fill in missing metadata fields, such as dates or author names, in documents to ensure accurate categorization and analysis.
36 #
Anomaly Detection
Anomaly Detection is a data mining technique used to identify patterns or data p… #
Anomalies may indicate errors, fraud, or unusual events that require further investigation.
Example #
In a legal document review, anomaly detection may be used to identify documents that contain unusual language patterns, attachments, or metadata that do not conform to typical document characteristics.
37 #
Hyperparameter Tuning
Hyperparameter Tuning is the process of optimizing the hyperparameters of a mach… #
Hyperparameters control the behavior of the model and are adjusted through experimentation and validation.
Example #
In a legal document review, hyperparameter tuning may involve adjusting the learning rate, regularization parameters, or model architecture of a predictive coding model to maximize its accuracy and efficiency.
38 #
Grid Search
Grid Search is a hyperparameter tuning technique that involves systematically se… #
Grid search helps automate the optimization process and find the optimal hyperparameters.
Example #
In a legal document review, grid search may be used to test different combinations of hyperparameters for a predictive coding model and select the ones that result in the highest accuracy and recall.
39 #
Model Selection
Model Selection is the process of choosing the best predictive model or algorith… #
Model selection involves evaluating multiple models and selecting the one that best fits the data and produces accurate predictions.
Example #
In a legal document review, model selection may involve comparing the performance of different predictive coding models, such as logistic regression, random forest, or support vector machines, and selecting the one that achieves the highest accuracy and precision.
40 #
Data Leakage
Data Leakage refers to the accidental or intentional exposure of information in… #
Data leakage can lead to biased results, overfitting, or inaccurate predictions.
Example #
In a legal document review, data leakage may occur if documents from one case are inadvertently included in the training set for a predictive coding model designed for another case, leading to incorrect classifications.
41 #
Model Evaluation Metrics
Model Evaluation Metrics are quantitative measures used to assess the performanc… #
Common evaluation metrics include accuracy, precision, recall, F1 score, area under the ROC curve, and confusion matrix.
Example #
In a legal document review, model evaluation metrics are used to compare the performance of different predictive coding models and select the one that achieves the highest accuracy and reliability.
42 #
Feature Selection
Feature Selection is the process of identifying and selecting the most relevant… #
Feature selection helps reduce dimensionality, improve model performance, and interpret the results.
Example #
In a legal document review, feature selection may involve identifying key metadata fields, such as sender, recipient, and date, that are most predictive of document relevance and should be used in the predictive coding model.
43 #
Natural Language Processing (NLP)
Natural Language Processing (NLP) is a branch of artificial intelligence that fo… #
Natural Language Processing (NLP) is a branch of artificial intelligence that focuses on