LLM Document Classification Using Reference Questions at Scale
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document classification methods, such as TAR and CAL, are time-consuming, prone to human bias, and require significant human intervention, making them inefficient for large volumes of unstructured content.
Innovation Solution
A method utilizing two large language models (LLMs) to generate and answer reference questions, where a first LLM generates questions based on reference documents and a second LLM assesses candidate documents' relevance, eliminating the need for human intervention and reducing bias.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If TAR (Technology Assisted Review) is used to classify documents, then classification accuracy can be improved through machine learning models, but the process becomes highly time-consuming due to multiple iterations requiring human input
Solution Approach 1:
The patent applies preliminary action by pre-generating reference questions from reference documents before classifying candidate documents. The first LLM generates questions about reference documents in advance, which are then used to automatically classify candidate documents without requiring iterative human feedback. This eliminates the time-consuming iteration process while maintaining classification accuracy through the pre-computed reference questions.
2Measurement precision
If TAR with multiple iterations is used to train classification models, then model performance can improve, but human bias and variability are introduced through subjective human input
Solution Approach 1:
The patent applies self-service by enabling the system to automatically generate reference questions and classify documents without requiring human intervention. The first LLM generates reference questions from reference documents, and the second LLM uses these questions to classify candidate documents. This automated self-service process eliminates human bias and variability while maintaining consistent classification performance across all documents.
Solution Approach 2:
The patent uses an intermediary approach by introducing reference questions as a mediator between reference documents and candidate documents. Instead of directly comparing documents or requiring human judgment, the reference questions serve as an intermediate representation that captures the essential classification criteria. This intermediary mechanism ensures consistent and objective classification by relying on the LLM-generated questions rather than subjective human input.
3Adaptability or versatility
If CAL (Continuous Active Learning) is used to accumulate training data from operator decisions, then the system can learn from human input, but it requires significant ongoing human intervention
Solution Approach 1:
The patent applies self-service by enabling the system to automatically generate reference questions and perform document classification without requiring continuous human intervention. The first LLM generates reference questions from reference documents, and the second LLM classifies candidate documents using these questions. This automated process eliminates the need for ongoing human input in CAL while maintaining the system's ability to adapt to different document types and classification criteria through the LLMs' learning capabilities.
Data Source
AI summary
A method for sorting candidate documents into several sets associated with a reference document, each document stored by a client device memory, wherein the method includes a device processor performing for each reference document, generating a first prompt for a first large language model requesting generation of at least one question determining the relevance of a candidate document to a reference document; (b) for each candidate document, generating at least one second prompt for a second large language model requesting the answer to at least one reference question; assigning each candidate document to the set associated with a reference document as a function of the value(s) that has been received for a second prompt containing a reference question associated to the reference document.