Active Learning Document Ranking for Legal Discovery

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The overwhelming process of reviewing thousands of documents in legal discovery is inefficient due to the lack of effective methods for identifying relevant information, leading to a need for improved document filtering and ranking techniques.

Innovation Solution

A computer-implemented method using a machine-learning model to predict the relevancy of documents to a specific issue, with user feedback for model parameter adjustment and document reranking, along with a user interface for document verification and analysis.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual review of all documents is performed, then comprehensive identification of relevant information is achieved, but time consumption and workload increase significantly

Engineering Contradiction:

Improveidentification accuracyVSAvoidreview time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system enables self-service through automated machine learning models that independently analyze and rank documents without requiring manual review of every document. The model continuously learns from user feedback and automatically improves its identification accuracy while reducing the time burden on users.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces the mechanical manual review process with an automated machine learning-based sorting system. The ML model automatically analyzes document content, assigns relevancy scores, and ranks documents, substituting human manual sorting efforts with computational automation that maintains high accuracy while dramatically reducing review time.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If machine learning model is used for document ranking, then review efficiency is improved, but model accuracy may be insufficient without continuous training

Engineering Contradiction:

Improvereview efficiencyVSAvoidprediction accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system implements continuous feedback loops where user corrections and validations of model predictions are fed back into the training dataset. The ML model is retrained periodically with this new data, allowing it to continuously improve accuracy while maintaining high review efficiency. This feedback mechanism ensures the model adapts to evolving document patterns and user preferences.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent employs dynamic model training where the ML system transitions from static initial training to continuous adaptive learning. The model's parameters and structure can evolve over time based on incoming data and performance metrics, allowing it to dynamically adjust to new document types, issues, and user behaviors while maintaining consistent productivity gains.

Inventive Principle:

Principle #15Dynamics

3Measurement precision

If comprehensive document analysis is performed, then relevancy identification accuracy is improved, but computational resources and processing time increase

Engineering Contradiction:

Improverelevancy prediction accuracyVSAvoidcomputational resource consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary analysis by pre-processing documents and extracting key features before the main ML classification step. Documents are pre-ranked using efficient filtering techniques, and the ML model focuses computational resources only on borderline cases or documents requiring detailed analysis, reducing overall computational energy consumption while maintaining high prediction accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent applies local quality by using different analysis depths for different documents. High-confidence documents receive minimal processing, while uncertain documents undergo more comprehensive analysis. The system dynamically adjusts the level of computational effort applied to each document based on its characteristics and the model's initial assessment, optimizing the balance between accuracy and resource usage.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS11520844B2Continuous learning, prediction, and ranking of relevancy or non-relevancy of discovery documents using a caseassist active learning and dynamic document review workflow

Publication Date: 2022.12.06 CASEPOINT LLC

AI summary

A method includes receiving a set of documents associated with data discovery. The method further includes receiving, for each document in a subset of the set of documents, an indication of relevancy or non-relevancy of the document for an issue. The method further includes modifying one or more parameters for a machine-learning model based on the indication of relevancy or non-relevancy. The method further includes outputting, for each document in the set of documents, by a machine-learning model, a prediction probability of relevancy to an issue associated with the data discovery, and a ranking of the set of documents based on the prediction probability of relevancy. The method further includes generating a user interface that includes a sampling of the documents for review by a user, where each document is associated with a predicted relevancy tag or a predicted non-relevancy tag.