Self-Contrastive Decorrelation for Sentence Embedding Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The generation of negative pairs for training machine learning models to determine sentence embeddings is challenging, memory-intensive, and requires a large dataset.
Innovation Solution
The self-contrastive decorrelation (SCD) technique trains machine learning models by performing self-contrasts on augmentations of a single sentence, eliminating the need for explicit negative pairs and reducing the required dataset size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If negative pairs are used for training machine learning models, then the model can learn sentence embeddings with accurate similarity discrimination, but the training process becomes memory-intensive and requires large datasets
Solution Approach 1:
The model performs self-contrastive learning by generating multiple augmented views of the same sentence through different encoding operations with varying dropout ratios. The sentence embedding is computed from these self-generated views without requiring external negative pairs, making the training process self-sufficient and reducing dependency on large datasets
Solution Approach 2:
The patent transforms the training approach by moving from comparing embeddings in the original vector space to comparing them in a decorrelated high-dimensional space. This dimensional transformation enables more effective contrastive learning with fewer samples by creating orthogonal feature representations that capture sentence semantics more efficiently
2Measurement precision
If negative pairs are used for training machine learning models, then the model can learn sentence embeddings with accurate similarity discrimination, but the training process becomes memory-intensive
Solution Approach 1:
The patent extracts and removes the negative pairs from the training process entirely, keeping only the positive contrastive pairs (different augmented views of the same sentence). This extraction eliminates the memory burden of storing and processing negative pairs while retaining the essential contrastive learning mechanism
Solution Approach 2:
The training process is segmented into distinct encoding operations with different dropout ratios, where each operation generates a specific augmented view. This segmentation allows the model to learn from multiple perspectives of the same sentence without requiring multiple complete sentence pairs, reducing computational overhead
3Measurement precision
If negative pairs are used for training machine learning models, then the model can learn sentence embeddings with accurate similarity discrimination, but the training process becomes challenging to generate automatically
Solution Approach 1:
The model generates its own training contrasts automatically by applying different encoding operations with varying dropout ratios to the input sentence. This self-service mechanism eliminates the need for manual or automated generation of negative pairs, making the training data preparation trivial and automatic
Solution Approach 2:
The patent performs preliminary augmentation of the input sentence through multiple encoding operations with different dropout ratios before computing the final embedding. This preliminary action creates the contrastive pairs needed for training automatically, eliminating the need for subsequent negative pair generation
Data Source
AI summary
A method for training a machine learning model using self-contrastive decorrelation is provided. The method comprises training a machine learning model by receiving a sentence including text, performing a first encoding operation on the sentence, performing a second encoding operation on the sentence, mapping the first vector representation on which a first augmentation operation is performed to a first high dimensional vector representation and the second vector representation on which a first augmentation operation is performed to a second high dimensional vector representation, generating a correlation matrix using the first high dimensional vector representation and the second high dimensional vector representation, and performing a decorrelation operation on the correlation matrix. The method includes receiving, by the trained machine learning model, an query that includes a target sentence, and outputting, using the trained machine learning model, a result sentence that satisfies a similarity metric relative to the target sentence.