Document Review System Using Relevancy Probability and Statistical Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current document review methods, particularly in legal discovery, are inefficient and costly due to the reliance on conventional search engine techniques that lack precision and recall, often returning irrelevant documents and missing relevant ones, especially with increasing document volumes.
Innovation Solution
Implementing a method that includes determining the probability of relevancy for documents using query terms, expanding search queries with synonyms, and applying statistical techniques to identify and add relevant documents from email threads and attachments, while ensuring a high threshold of relevance through automated processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional keyword search is used to retrieve documents, then the search process is simple and fast, but the precision and recall are low, returning many irrelevant documents and missing relevant ones
Solution Approach 1:
The patent introduces human reviewers as intermediaries between the search engine and the final document selection. The search engine retrieves candidate documents, then human reviewers assess their relevancy, and statistical sampling determines whether additional review is needed. This intermediary human judgment layer resolves the contradiction by providing high precision while maintaining reasonable productivity through selective review.
Solution Approach 2:
The patent changes the parameter of relevancy assessment from binary (relevant/not relevant) to probabilistic (confidence level). By using statistical sampling to calculate confidence levels and comparing them against thresholds, the system dynamically adjusts the review process to achieve both efficiency and accuracy, resolving the precision-speed contradiction.
2Measurement precision
If all documents are reviewed manually to ensure high precision, then the accuracy of identifying relevant documents is high, but the time and cost required are prohibitively expensive
Solution Approach 1:
The patent applies partial review action by using statistical sampling to determine the minimum necessary review effort. Instead of reviewing all documents, the system calculates a confidence level based on sampling and only reviews additional documents if the confidence threshold is not met. This partial action approach achieves high precision while minimizing time loss.
Solution Approach 2:
The patent implements feedback loops where reviewers assess documents and provide information that updates the confidence level calculation. This feedback mechanism allows the system to adaptively determine when sufficient review has been performed, balancing precision requirements with time constraints by stopping review when confidence thresholds are achieved.
3Productivity
If statistical sampling is used to reduce review workload, then the efficiency and cost effectiveness improve, but the confidence in completeness of relevant documents may be reduced
Solution Approach 1:
The patent uses feedback from statistical sampling results to dynamically adjust the review process. The confidence level calculated from sampling acts as feedback that determines whether additional review is needed. If confidence is below the threshold, the system retrieves and reviews more documents, thereby maintaining reliability while maximizing efficiency through selective application of full review.
Solution Approach 2:
The patent performs preliminary statistical sampling before committing to full review of all documents. This preliminary action provides early information about document relevancy and confidence levels, allowing the system to avoid unnecessary full reviews while maintaining reliability through threshold-based decision-making about when additional review is required.
4Measurement precision
If synonyms and phrase variations are included in search queries to improve recall, then more relevant documents are found, but the number of documents to review increases significantly
Solution Approach 1:
The patent uses human reviewers as intermediaries to handle the increased document volume resulting from synonym-based searches. Rather than relying solely on automated filtering of the expanded result set, human judgment assesses relevancy of documents retrieved with varied search terms, maintaining high recall while managing the quantity of documents through selective human review guided by confidence thresholds.
Data Source
AI summary
Methods for reviewing a collection of documents to identify relevant documents from the collection are provided. A search of the collection can be run based on query terms, to return a subset of responsive documents. A probability of relevancy can be determined for a document in the returned subset, and the document is removed from the subset if it does not reach a threshold probability of relevancy. Documents in a thread of a correspondence (for example, an e-mail) in the responsive documents subset can be added to the responsive documents subset. Further, an attachment to a document in the responsive documents subset can be added to the responsive documents subset. A statistical technique can be applied to determine whether remaining documents in the collection meet a predetermined acceptance level.