NLP Model External Knowledge Base Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Pre-trained language models (PLMs) face challenges in accurately retrieving factual knowledge due to implicit representation of knowledge as parameter weights, which limits the amount of information stored and makes it difficult to retrieve relevant information effectively.
Innovation Solution
Retrieving information from a knowledge base and inputting it into the NLP model to generate responses, using identified entities to retrieve relevant triples with similarity scores to select the most relevant information, allowing the model to process both natural language input and knowledge base triples to improve accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If factual knowledge is implicitly represented as parameter weights within an NLP model, then the model can process natural language tasks, but the amount of information that can be stored is limited and retrieval accuracy deteriorates
Solution Approach 1:
The patent segments the knowledge storage function from the NLP model by introducing an external knowledge base that stores factual knowledge separately. The model parameters focus on linguistic processing while the knowledge base handles factual information, allowing both to optimize their respective functions without competing for the same parameter space.
Solution Approach 2:
The patent introduces an intermediary retrieval mechanism that connects the NLP model to the external knowledge base. This intermediary system translates model predictions into knowledge base queries and processes the retrieved information, enabling accurate access to stored factual knowledge without requiring the model to implicitly encode all facts.
2Adaptability or versatility
If more factual knowledge is stored in NLP model parameters, then the model has more world knowledge, but training data requirements and model complexity increase
Solution Approach 1:
The patent extracts the factual knowledge storage function from the NLP model and places it in an external knowledge base. This separation allows the model to maintain relatively simple architecture focused on linguistic understanding, while the knowledge base handles the burden of storing extensive factual information about entities, relationships, and attributes.
Solution Approach 2:
The external knowledge base serves as a universal repository that can store diverse factual knowledge across multiple domains and entity types. This single knowledge base structure can accommodate various kinds of factual information (attributes, relationships, events) that would otherwise require separate model components or extensive training data.
3Loss of information
If the NLP model is trained on large amounts of unstructured text, then linguistic nuances are captured, but the implicit knowledge storage leads to inaccurate model predictions
Solution Approach 1:
The patent segments the knowledge representation into two distinct components: linguistic patterns captured in the model parameters from unstructured text training, and explicit factual knowledge stored in the external knowledge base. This segmentation allows the model to retain its ability to understand linguistic nuances while relying on the structured knowledge base for accurate factual predictions.
Solution Approach 2:
The system incorporates feedback mechanisms where the model's predictions are checked against the external knowledge base, and the retrieved knowledge is used to correct or refine model outputs. This feedback loop ensures that predictions are both linguistically appropriate (from model training) and factually accurate (from knowledge base verification).
Data Source
AI summary
The technology described herein receives a natural-language sequence of words comprising multiple entities. The technology then identifies a plurality of entities in the natural-language sequence. The technology generates a masked natural-language sequence by masking a first entity in the natural-language sequence. The technology retrieves, from a knowledge base, information related to a second entity in the plurality of entities. The technology then trains a natural-language model to respond to a query. The training uses a first representation of the masked natural-language sequence, a second representation of the information, and the first entity.