Dual Encoder LLM Training With Unique IDs for Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) trained on pre-processed embeddings struggle to effectively incorporate novel task types outside retrieval and similarity/relatedness matching tasks, leading to misidentification of positive examples as negative examples in classification tasks.
Innovation Solution
Assigning unique identifiers to pairs of input text and classification labels, using a random number generator or fingerprint, to distinguish correct targets in dual encoder embeddings models, allowing seamless integration of classification tasks without modifying the models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dual encoder embeddings models are trained on classification tasks without unique identifiers, then the model can process classification data, but positive examples are misidentified as negative examples when multiple inputs share the same classification label
Solution Approach 1:
The patent segments the training data by assigning unique identifiers to each input-text classification-label pair, allowing the model to distinguish between different positive examples that share the same classification label. This segmentation prevents the model from confusing different positive examples as negative examples during training.
Solution Approach 2:
The unique identifier acts as an intermediary element between the input text and classification label, providing an additional dimension of information that helps the model correctly identify positive examples. The identifier mediates the relationship between text and label, ensuring that multiple inputs with the same label are treated as distinct positive examples rather than negatives.
2Reliability
If the model architecture is modified to support novel task types, then classification performance improves, but compatibility with existing retrieval and similarity training mixtures is lost
Solution Approach 1:
The patent implements a universal training framework where the same dual encoder embeddings model architecture can handle multiple task types (retrieval, similarity, and classification) by using task-specific training mixtures. The model remains adaptable to different tasks without requiring architectural modifications, as the unique identifier mechanism works within the existing model framework.
Solution Approach 2:
The patent prepares classification data in advance by assigning unique identifiers to each input-text classification-label pair before training. This preliminary preparation allows the model to process classification tasks using the same architecture and training mechanisms as retrieval tasks, maintaining compatibility while improving performance.
3Reliability
If classification examples are formatted to look like retrieval problems, then model performance improves, but the training data does not directly target embedding representations for classification tasks
Solution Approach 1:
The patent applies local quality by assigning unique identifiers specifically to classification examples while maintaining the general retrieval problem format for other examples. This allows the training data to have different characteristics for different task types, with classification examples receiving the additional identifier treatment that directly improves embedding representation quality for classification tasks.
Data Source
AI summary
Aspects of the disclosure relate to pre-format embeddings used for model training, and in particular, training of dual encoder LLMs. For instance, a dataset batch of pairs of input text and one or more classification labels may be accessed. A unique identifier to each pair of input text and one or more classification labels may be assigned. The pairs of input text and one or more classification labels and assigned unique identifiers may be used to train a model to assign classification labels to textual inputs.