Generative AI Prompt Refinement for Consistent Document Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional eDiscovery processes face inefficiencies and inconsistencies due to the need for manual review and varying interpretations of document relevance by different attorneys, leading to conflicting classifications and the cumbersome training of machine learning classifiers.
Innovation Solution
A generative AI model is used to iteratively refine prompts based on prompt criteria, evaluating classification performance and updating prompts to improve document classification accuracy, reducing the need for extensive manual review and aligning different reviewer interpretations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If machine learning models are deployed for document classification, then classification speed is improved, but classification consistency deteriorates due to different attorneys having different interpretations
Solution Approach 1:
The system implements an iterative feedback loop where classification results are evaluated against prompt criteria, and prompts are updated based on performance metrics. This continuous feedback mechanism ensures that the classification model aligns with legal standards and maintains consistency across different reviewers' interpretations.
Solution Approach 2:
The system dynamically adjusts prompt parameters and criteria based on evaluation results. By modifying prompt formulations and classification thresholds in response to performance data, the system adapts to different legal interpretations while maintaining standardized classification behavior.
2Measurement precision
If manual review of documents is performed to train classifiers, then classification accuracy is improved, but time consumption increases due to the need to review thousands of documents
Solution Approach 1:
The system performs preliminary actions by pre-defining comprehensive prompt criteria that encapsulate legal standards before actual classification begins. This preliminary preparation eliminates the need for manual review of thousands of documents during training, as the criteria are established in advance through structured prompt engineering.
Solution Approach 2:
The system enables self-service classification where the model uses pre-established prompt criteria to autonomously evaluate document relevance without requiring extensive manual training data. The model serves itself by applying learned criteria consistently across all documents, reducing the need for human review while maintaining accuracy.
3Adaptability or versatility
If multiple attorneys review documents with different interpretations, then comprehensive coverage is improved, but conflicting classifications increase
Solution Approach 1:
The system creates a universal classification framework through standardized prompt criteria that can be applied across different legal contexts and by different reviewers. This multi-functional prompt structure enables consistent classification behavior while maintaining adaptability to various legal interpretations and document types.
Data Source
AI summary
The following relates generally to using generative AI to: (i) classify documents; (ii) generate prompts to classify documents; (iii) evaluate the classification performance of prompts; (iv) generate updates to prompts; and/or (v) evaluate the classification performance of updated prompts. In some embodiments, one or more processors: obtain prompt criteria associated with a corpus of documents, wherein the prompt criteria defines at least a relevancy requirement for an inquiry and a description of an issue; generate a prompt for input into a generative AI model based upon the prompt criteria; evaluate classification performance of the prompt at classifying documents with respect to the relevancy requirement; obtain an updated prompt criteria including an updated description of the issue; generate an updated prompt; evaluate classification performance of the updated prompt at classifying documents with respect to the relevancy requirement; and based on the evaluation, approve the updated prompt to classify additional documents.