Prompt-Based Active Learning Queues for Consistent Document Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing eDiscovery processes face inefficiencies and inconsistencies due to the need for significant manual review to train machine learning classifiers and varying interpretations by attorneys, leading to conflicting document classifications.
Innovation Solution
A prompt-based classification model using generative AI is employed, where prompts are refined to classify documents, with feedback from reviewers integrated to update the model, reducing manual review requirements and enhancing classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If machine learning models are deployed for document classification, then classification speed is improved, but classification consistency deteriorates due to varying attorney interpretations
Solution Approach 1:
The system implements feedback mechanisms where reviewers provide corrections on misclassified documents, and this feedback is used to retrain and refine the classification model. This continuous feedback loop ensures the model adapts to varying interpretations while maintaining consistency across different attorneys' workstreams.
Solution Approach 2:
The system dynamically adjusts classification parameters and thresholds based on feedback data and reviewer behavior patterns. By changing model parameters adaptively rather than using fixed thresholds, the system maintains high classification speed while improving consistency across different legal interpretations.
2Measurement precision
If manual review of documents is performed to train classifiers, then classification accuracy is improved, but time consumption increases
Solution Approach 1:
Instead of requiring complete manual review of all documents for training, the system uses partial review of a strategically selected subset of documents. The system identifies and reviews only the most informative documents for training purposes, achieving sufficient accuracy without the time cost of reviewing every document manually.
Solution Approach 2:
The system performs self-service by automatically generating training data and classification models with minimal human intervention. Reviewers only need to correct obvious errors, and the system automatically processes the majority of classification tasks, dramatically reducing manual review time while maintaining accuracy.
3Adaptability or versatility
If multiple attorneys use different machine learning models, then flexibility in classification is improved, but conflicts in document indication increase
Solution Approach 1:
The system creates a universal classification model that can be used across all attorneys' workstreams. This single model serves multiple functions and attorneys simultaneously, eliminating the need for separate models for each attorney and thereby preventing conflicting document indications while preserving the flexibility to handle different classification scenarios.
Data Source
AI summary
The following relates generally to using generative AI to: (i) classify documents; (ii) generate prompts to classify documents; (iii) evaluate the classification performance of prompts; (iv) generate updates to prompts; and/or (v) train classifiers. In some embodiments, one or more processors: generate a prompt for input to the generative AI model; generate classifications for a set of documents from the corpus of documents by inputting the set of documents and the prompt to the generative AI model; based on the classifications, provide the set of documents to a review platform for manual review by a reviewer; obtain review data associated with a subset of documents from the set of documents; and train, by executing a training algorithm, a classifier using the review data as ground truth data, wherein the training algorithm is configured to analyze extracted relevant document portions of the subset of documents to train the classifier.