Asymmetric Dual Encoder With Shared Projection Layer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Dual encoder architectures in natural language processing for question answering tasks face performance issues due to parameter sharing strategies, which affect the alignment of question and answer embeddings, leading to suboptimal retrieval quality.
Innovation Solution
The implementation of an asymmetric dual encoder system with a shared projection layer, where token embedders and encoders can be distinctly parameterized, and initialized from a common set of pre-trained parameters, optimized using contrastive loss with in-batch sampled soft-max, enhances the alignment of question and answer embeddings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If parameter sharing is implemented across both encoders in a dual encoder arrangement, then model performance and retrieval quality are improved, but device complexity and computational overhead increase
Solution Approach 1:
The patent segments the parameter sharing strategy by applying it only to the projection layer rather than the entire encoder architecture. This allows the benefits of parameter sharing (improved retrieval quality through better embedding alignment) while limiting the complexity increase to only the projection layer, not the full encoder depth.
Solution Approach 2:
The patent applies parameter sharing locally only where most beneficial - in the projection layer that directly impacts embedding alignment and retrieval quality. The encoder layers maintain distinct parameters to avoid excessive complexity, while the projection layer uses shared parameters to ensure consistent embedding space alignment for both queries and documents.
2Device complexity
If distinct parameterization is used for both encoders, then device complexity is reduced and training is simplified, but embedding alignment and retrieval quality deteriorate
Solution Approach 1:
The patent divides the model into two segments with different parameter strategies: encoder layers use distinct parameters for simplicity and reduced complexity, while the projection layer uses shared parameters to ensure proper embedding alignment and maintain high retrieval quality.
Solution Approach 2:
The patent applies different parameter strategies to different parts of the model based on their specific functions. The encoder layers use distinct parameters where complexity reduction is beneficial, while the projection layer uses shared parameters where embedding alignment quality is critical for retrieval performance.
3Reliability
If parameter sharing is implemented in the projection layer, then embedding alignment and retrieval quality are improved, but training time and computational resources increase
Solution Approach 1:
The patent limits parameter sharing to only the projection layer, which is a relatively small component compared to the full encoder. This segmentation approach allows retrieval quality improvement through projection layer alignment while avoiding the excessive training time that would result from sharing parameters across the entire encoder architecture.
Data Source
AI summary
Aspects of the technology provide systems and methods for implementing an asymmetric dual encoder architecture. The architecture includes a token embedder layer section having a first token embedding section associated with a first input and a second token embedding section associated with a second input, and an encoder layer section having a first encoder section receiving token embeddings from the first token embedding section and a second encoder section receiving token embeddings from the second token embedding section. A shared projection layer receives encodings from both the first and second encoder sections and generates a set of projections. An embedding space is configured, based on the set of projections, to generate a question embedding and an answer embedding, in which the question and answer embeddings are used in identifying a set of candidate answers to an input answer.