Dual Encoder Training with Target Embedding Drift Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training dual encoder models with large datasets is computationally challenging due to the impracticality of exhaustively recalculating target embeddings at each iteration, leading to accuracy drift and inefficient use of computational resources.
Innovation Solution
A correction model is employed to correct approximated target embeddings at each training iteration, reducing the need for exhaustive re-encoding and minimizing computational overhead, while maintaining accuracy by accounting for drift.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If target embeddings are exhaustively recalculated at each training iteration, then accuracy is maintained, but computational resources and time consumption become impractical
Solution Approach 1:
The patent pre-calculates target embeddings once and stores them in a buffer, avoiding the need to recalculate them at every training iteration. This preliminary action maintains accuracy while significantly improving training efficiency by reusing the pre-computed embeddings.
Solution Approach 2:
The patent creates a buffer copy of target embeddings that can be accessed during training without recalculating the original embeddings. This copying approach allows the system to maintain embedding accuracy while reducing computational overhead during each training iteration.
2Productivity
If target embeddings are cached without correction, then computational resources are saved, but accuracy drift occurs
Solution Approach 1:
The patent introduces a correction model that receives feedback about the drift in cached embeddings and adjusts them accordingly. This feedback mechanism allows the system to maintain both efficiency (by using cached embeddings) and accuracy (by correcting for drift).
Solution Approach 2:
The correction model modifies the embedding parameters by applying correction terms to the cached embeddings. This parameter change approach allows the system to maintain accuracy without recalculating the entire embedding, thus preserving computational efficiency.
3Reliability
If dual encoder model is trained with large datasets, then retrieval performance improves, but computational complexity becomes intractable
Solution Approach 1:
The patent segments the computational task by separating the embedding calculation (done once) from the training process (can use cached embeddings). This segmentation allows the system to handle large datasets effectively without the computational complexity of recalculating all embeddings at each iteration.
Solution Approach 2:
By performing the computationally intensive embedding calculation as a preliminary action before training, the system can then focus computational resources on the training process itself, making it feasible to work with large datasets without overwhelming computational complexity.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for that can train a dual encoder model using a correction model to correct target embeddings at each training iteration without explicitly recalculating each target embedding. In one aspect, a system comprises obtaining approximated target embeddings for a plurality of target data items, processing the respective approximated target embeddings using a correction model to generate corrected target embeddings, processing a query data item using a query encoder model to generate a query embedding, electing, using the corrected target embeddings and the query embedding, a subset of the target data items as relevant target data items, and training the dual encoder model on a loss function for the retrieval task using the relevant target data items for the one or more query data items.