Prompt Self-Optimization for Consistent Document Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing eDiscovery processes face inefficiencies and inconsistencies due to the need for manual review and varying interpretations of document classification by different attorneys, leading to conflicting results and cumbersome machine learning model deployment.
Innovation Solution
A generative AI model is used to generate and refine prompts for document classification, allowing for automated classification with reduced manual review, and includes feedback mechanisms to reconcile different user inputs and improve classification performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If machine learning models are deployed for document classification, then classification automation is improved, but model deployment complexity and training requirements increase
Solution Approach 1:
The patent extracts the complex machine learning model training and deployment requirements by using prompt-based classification with generative AI. Instead of training custom models, the system uses pre-trained models with dynamically generated prompts that encode classification criteria, thereby removing the complexity of model training while maintaining automation capabilities
Solution Approach 2:
The system changes the approach from model training parameters to prompt engineering parameters. By modifying the prompt inputs rather than training model weights, the system achieves reconfigurability without the computational and temporal costs of retraining, thus reducing deployment complexity while maintaining high automation
2Adaptability or versatility
If different attorneys use different machine learning models for classification, then classification flexibility is improved, but result consistency and reliability deteriorate
Solution Approach 1:
The patent implements a universal prompt-based framework that can be adapted to different classification needs by changing prompts rather than models. This allows different attorneys to use the same base system with customized prompts, ensuring consistent methodology while maintaining flexibility for different classification criteria
Solution Approach 2:
The system incorporates feedback mechanisms where classification results can be reviewed and corrected. This feedback loop ensures that regardless of which attorney uses the system, the final classifications align with expected standards, thereby improving reliability and consistency while preserving the flexibility to handle different case-specific requirements
3Measurement precision
If manual review of thousands of documents is performed for training, then classifier performance is improved, but time consumption and labor requirements increase
Solution Approach 1:
The patent applies preliminary action by pre-training the generative AI model and pre-defining classification frameworks. This allows the system to achieve high performance without requiring manual review of thousands of documents during each deployment, as the foundational model and evaluation criteria are established in advance
Solution Approach 2:
Instead of manually training models by copying and analyzing thousands of documents, the system uses prompt-based copying where classification criteria are encoded in prompts that can be rapidly replicated and modified. This maintains high classification performance while dramatically reducing the time and labor required for training and deployment
Data Source
AI summary
The following relates generally to using generative AI to: (i) classify documents; (ii) generate prompts to classify documents; (iii) evaluate the classification performance of prompts; (iv) generate updates to prompts; and/or (v) evaluate the classification performance of updated prompts. In some embodiments, one or more processors: obtain prompt criteria associated with a corpus of documents, wherein the prompt criteria defines at least a relevancy requirement for an inquiry and a description of an issue; generate a prompt for input into a generative AI model based upon the prompt criteria; evaluate classification performance of the prompt at classifying documents with respect to the relevancy requirement; obtain an updated prompt criteria including an updated description of the issue; generate an updated prompt; evaluate classification performance of the updated prompt at classifying documents with respect to the relevancy requirement; and based on the evaluation, approve the updated prompt to classify additional documents.