Language Model Labeling With Consensus and Dynamic Class Prompts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The manual labeling of items for downstream uses is time-consuming, resource-intensive, and prone to errors, making it cumbersome and expensive.
Innovation Solution
A computer-implemented technique that uses a machine-trained language model to automatically label items, allowing for dynamic task description adjustments, integration of human agents, and consensus-based classification, enabling efficient and consistent labeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual labeling is used to label items, then labeling accuracy can be maintained through human judgment, but the process consumes considerable time and resources
Solution Approach 1:
The patent introduces a machine-trained language model as an intermediary between the item to be labeled and the final label. The language model processes items automatically, providing labels that can then be reviewed or directly used, thereby reducing manual intervention time while maintaining accuracy through the model's trained capabilities
Solution Approach 2:
The patent employs preliminary action by using the language model to pre-label items before final human review or direct deployment. This pre-labeling process prepares the data in advance, reducing the time required for complete labeling while maintaining quality through subsequent verification steps
2Stability of the object's composition
If manual labeling is used to ensure consistent labeling quality, then labeling consistency can be maintained, but the process becomes expensive and resource-intensive
Solution Approach 1:
The patent implements self-service by enabling the language model to autonomously perform the labeling task without requiring extensive human intervention. The model consistently applies learned patterns to label items, maintaining labeling consistency while eliminating the need for large numbers of human labelers
Solution Approach 2:
The patent replaces the mechanical system of human manual labeling with an automated language model system. This substitution maintains consistency through the model's deterministic processing while dramatically reducing the quantity of human resources required
3Productivity
If a single language model is used for labeling, then processing speed is maintained, but the reliability of labeling results may be insufficient
Solution Approach 1:
The patent applies local quality by assigning different weights to different language models based on their performance characteristics for specific labeling tasks. This allows the system to optimize for both speed and reliability by selecting and weighting models according to their local strengths in different contexts
4Productivity
If the task description is fixed for all items, then processing efficiency is maintained, but the adaptability to different item types is limited
Solution Approach 1:
The patent implements dynamics by making the task description adjustable and adaptable based on the specific item being processed. The system can modify task descriptions dynamically to suit different item types while maintaining efficient processing through automated adaptation rather than fixed rigid templates
Data Source
AI summary
A computer-implemented labeling technique generates a task description that describes a labeling task to be given to a language model. The technique then sends a prompt to the language model, which includes the task description and a particular item to be labeled. The technique receives a response provided by the language model in response to the prompt, which specifies a class assigned by the language model to the item. In some implementations, the task description specifies a group of suggested classes to be used in classifying the particular item. The task description also invites the language model to specify another class upon a finding that none of the group of suggested classes applies to the item. The technique also allows a user to stop and restart a labeling run at any point in the labeling run. Other aspects of the technique include consensus processing and weight updating.