Corpus Vector Indexing for QA Retrieval Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current question answering systems, particularly generative systems, face limitations in response accuracy due to the quality and quantity of training samples and rely heavily on large corpora with text co-occurrence attributes, leading to inefficiencies in storage and retrieval of semantic information.
Innovation Solution
A corpus generating method that creates a corpus with an inverted chain index by generating corpus vectors based on content, determining vector types through Product Quantization, and clustering, allowing for efficient search and improved response accuracy by reflecting semantic similarity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If retrieval based question answering systems use text inverting method to recall candidate questions from corpus, then implementation is simple and accuracy is high, but the system requires very large corpus with abundant information and high storage burden
Solution Approach 1:
The patent transforms the corpus representation by converting text data into vector data through embedding models. This parameter change allows the system to work with compact vector representations instead of large text corpora, reducing storage requirements while maintaining the ability to perform semantic similarity searches for accurate answer retrieval
Solution Approach 2:
The patent creates vector copies of the corpus content that can be efficiently stored and searched. Instead of storing and searching the original large text corpus, the system stores compact vector representations that capture the semantic meaning, enabling fast retrieval without requiring the full text corpus in memory
2Extent of automation
If generative question answering systems use deep learning models trained on massive dialogue data, then automatic response generation is achieved, but response accuracy is limited by model quality and training data
Solution Approach 1:
The patent introduces a retrieval-based intermediary layer between user input and response generation. Instead of relying solely on generative models, the system first retrieves accurate answers from the corpus using vector similarity search, then uses the generative model to format the retrieved answer. This intermediary retrieval step ensures high accuracy by grounding responses in actual corpus content
Solution Approach 2:
The patent segments the question answering process into distinct stages: vector embedding, similarity search, candidate retrieval, and response generation. This segmentation allows the system to optimize each stage independently, using efficient vector search for retrieval and generative models only for formatting, thereby improving overall accuracy while maintaining automation
3Adaptability or versatility
If multiple pieces of data are stored for the same semantics in the corpus, then semantic coverage is improved, but storage burden and storage cost increase
Solution Approach 1:
The patent merges multiple pieces of data with the same semantics into a single vector representation. By converting text to vectors, the system can identify and merge redundant semantic content, storing only one vector per unique semantic meaning rather than multiple copies, thereby reducing storage burden while maintaining comprehensive semantic coverage
Data Source
AI summary
Embodiments of the present specification provide a corpus generating method and apparatus, and a human-machine interaction processing method and apparatus, wherein the corpus generating method comprises: generating a corpus vector according to corpus content; determining a vector type of the corpus vector; and generating, according to the vector type and the corpus vector, a corpus having an inverted chain index.