Sparse Transformer Matching for Scalable Question Answer Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for open-domain question answering struggle with scalability, speed, and accuracy, particularly in handling large volumes of text data, and existing systems are inflexible and lack explainability.
Innovation Solution
The implementation of Sparse Transformer Matching (SPARTA) that combines classic IR with transformer-based representation learning, using a trainable system for indexing and scoring, enabling efficient and accurate retrieval of answers from a large pool of documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If BERT-based MRC models are used for document-level QA, then answer accuracy is improved, but processing speed deteriorates (only a few thousand words per second)
Solution Approach 1:
The patent segments the QA task into two distinct stages: (1) a retriever stage that filters documents using sparse vector matching, and (2) an MRC reader stage that extracts answers using BERT-based models. This segmentation allows the system to leverage the speed of sparse matching for filtering while maintaining the accuracy of BERT-based models for final answer extraction, resolving the contradiction between accuracy and processing speed.
Solution Approach 2:
The patent applies preliminary action by performing sparse vector matching and document filtering before applying the computationally intensive BERT-based MRC models. The retriever stage pre-processes and narrows down the document pool to only those most relevant to the query, so that the expensive MRC models are applied to a small subset of documents, thereby maintaining accuracy while dramatically improving overall processing speed.
2Speed
If traditional IR-based retrievers are used to filter documents, then response speed is maintained, but performance is bounded and the system lacks trainability
Solution Approach 1:
The patent changes the parameters of the retriever by transitioning from traditional IR methods (BM25, TF-IDF) to a trainable embedding-based approach using sparse transformer matching. The system learns optimal embedding dimensions, sparsity levels, and matching parameters from data, enabling the retriever to be optimized for specific QA tasks while maintaining fast response times through efficient sparse vector operations.
3Measurement precision
If dense vector representations are used for matching, then retrieval accuracy may improve, but computational efficiency and scalability deteriorate
Solution Approach 1:
The patent extracts only the most relevant features from the embedding space by applying sparsity constraints. Instead of using full dense vectors, the system identifies and retains only the top-k most significant dimensions or tokens that contribute to matching quality, discarding redundant information. This extraction of essential features maintains retrieval accuracy while dramatically reducing computational complexity and improving efficiency.
Solution Approach 2:
The patent applies local quality by making different parts of the vector representation have different properties: the most important dimensions are retained with high weight, while less important dimensions are sparsified or zeroed out. This allows the system to concentrate computational resources on the most informative features, achieving high retrieval accuracy with improved computational efficiency through non-uniform quality distribution across vector components.
Data Source
AI summary
Systems and methods involving SPARTA (Sparse Transformer Matching) are disclosed. Embodiments may relate to scalable and transferable retrieval methods for question answering.