Prompt-Criteria Tuning for Consistent Generative AI Document Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing eDiscovery processes face inefficiencies and inconsistencies due to the need for manual review of large document corpora and varying interpretations by attorneys, leading to conflicting machine learning model outputs and cumbersome training requirements.
Innovation Solution
A generative AI model is used to classify documents through iterative prompting, where prompt criteria are refined based on review data, allowing for efficient and consistent document classification without extensive manual training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If machine learning models are deployed for document classification, then classification capability is provided, but training complexity and manual review requirements increase significantly
Solution Approach 1:
The patent replaces traditional machine learning training mechanisms with prompt-based generative AI. Instead of training models manually through extensive labeled data review, the system uses prompts to guide the AI model in classifying documents, substituting the complex training process with a simpler prompting and iteration mechanism.
Solution Approach 2:
The patent changes the approach from training model parameters through manual labeling to modifying prompt parameters through iteration. The system evolves prompts based on review data and classification results, transforming the optimization process from training data manipulation to prompt engineering, which reduces manual intervention requirements.
2Adaptability or versatility
If different attorneys use different machine learning models for classification, then diverse classification perspectives are achieved, but conflicting classifications and inconsistencies occur
Solution Approach 1:
The patent implements a universal prompt-based system that can be applied across different attorneys and cases. The same generative AI model with iterative prompting mechanism serves all classification needs, allowing the system to adapt to different legal contexts while maintaining consistent classification behavior through standardized prompt evolution processes.
Solution Approach 2:
The system incorporates feedback loops where classification results and review data are used to iteratively improve prompts. This feedback mechanism ensures that classification consistency is maintained across different attorneys by continuously aligning prompt behavior with actual review outcomes, resolving conflicts through data-driven prompt optimization.
3Measurement precision
If manual review of thousands of documents is performed for training, then sufficient labeled examples are obtained, but time consumption and labor requirements increase significantly
Solution Approach 1:
The patent enables the system to self-improve through iterative prompting. The generative AI model automatically generates classification results that are reviewed and used to refine prompts, creating a self-service optimization loop that eliminates the need for manual training of thousands of labeled examples while maintaining high classification accuracy.
Solution Approach 2:
The system performs preliminary classification using the generative AI model before human review. This preliminary action filters and pre-processes documents, reducing the amount of manual review needed while maintaining accuracy, as the AI model's initial classifications guide subsequent human verification and prompt optimization.
Data Source
AI summary
The following relates generally to using generative AI to: (i) classify documents; (ii) generate prompts (and/or criteria for prompts) to classify documents; (iii) explain document classifications; and/or (iv) explain updates to prompts (and/or prompt criteria). In some embodiments, one or more processors: obtain at least one prompt criteria defining context for classifying a corpus of documents using the generative AI model; generate a first prompt based upon the at least one prompt criteria; input the first prompt and a first document into the generative AI model to obtain a classification of the first document; obtain review data associated with the first document; update the at least one prompt criteria based on the classification of the first document and the review data; generate a second prompt based upon the updated at least one prompt criteria; and classify a second document by inputting the second prompt into the generative AI model.