Shared-Specialized Transformer Encoder for Passage Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Bi-encoder architectures for passage retrieval are inefficient in terms of resource consumption and sensitive to distribution shifts, leading to suboptimal performance in open domain question answering tasks.
Innovation Solution
A transformer-based text encoder with a shared transformer block and specialized transformer block is used, where the shared block is trained for both questions and passages, and the specialized block includes input-specific subnetworks for improved encoding efficiency and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If bi-encoder architectures are used for passage retrieval, then passage retrieval can be performed, but resource consumption increases and performance degrades due to sensitivity to distribution shifts
Solution Approach 1:
The patent merges the question encoder and passage encoder into a single unified transformer-based text encoder that processes both questions and passages through shared transformer blocks. This consolidation reduces the total number of parameters and computational resources required, while specialized transformer blocks handle input-specific transformations. The unified architecture eliminates the need for separate bi-encoder structures, thereby reducing resource consumption while maintaining retrieval accuracy.
Solution Approach 2:
The transformer-based text encoder is designed with multi-functionality to handle both question encoding and passage encoding within a single model. The shared transformer blocks are trained to process diverse input types (questions and passages) uniformly, while specialized transformer blocks adapt to input-specific characteristics. This universal encoder replaces the specialized bi-encoder architecture, reducing resource overhead and improving robustness to distribution shifts across different domains.
2Measurement precision
If separate encoders are used for questions and passages, then encoding can be specialized, but device complexity increases
Solution Approach 1:
The unified transformer-based text encoder is segmented into shared transformer blocks and specialized transformer blocks. The shared blocks handle common encoding tasks for both questions and passages, while specialized blocks handle input-specific transformations. This segmentation allows the model to maintain encoding precision for different input types without requiring completely separate encoder architectures, thereby reducing overall system complexity while preserving accuracy.
Solution Approach 2:
The encoder architecture applies local quality by having specialized transformer blocks that specifically handle question inputs and other blocks that handle passage inputs. Instead of making the entire encoder specialized for different input types, only specific blocks are specialized, while others remain shared. This localized specialization maintains high encoding accuracy for each input type without the full complexity of separate encoders.
Data Source
AI summary
A computing system includes a logic subsystem and a storage subsystem holding instructions executable by the logic subsystem to implement a transformer-based text encoder. The transformer-based text encoder includes a plurality of transformer blocks previously-trained to apply encoding operations to computer-readable text representations of input text strings, the computer-readable text representations including computer-readable question representations of input text questions, and computer-readable passage representations of input text passages. The plurality of transformer blocks include a shared transformer block trained for both the computer-readable question representations and the computer-readable passage representations and a specialized transformer block including two or more input-specific subnetworks, and a routing function to select an input-specific subnetwork of the two or more input-specific subnetworks for each of the computer-readable text representations.