Soft Knowledge Prompts for Updatable Language Model Memory Without Retraining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural language models face limitations such as fixed memory size, lack of control over data storage, and obsolescence of memorized knowledge, making them inefficient for knowledge-intensive tasks.
Innovation Solution
Employing soft knowledge prompts trained via self-supervised learning on knowledge bases to function as an external memory, allowing flexible expansion and updating without retraining the model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the language model uses parametric memory with fixed size, then the model can store factual information from pre-training data, but the memory size cannot grow or shrink over time without fully retraining the model
Solution Approach 1:
The patent divides the memory system into two separate components: a fixed-size language model for general language processing and a separate, dynamically adjustable knowledge base for storing factual information. This segmentation allows the knowledge base to be independently resized or updated without affecting or requiring retraining of the language model, thus resolving the contradiction between memory adaptability and retraining complexity.
Solution Approach 2:
The patent extracts the knowledge storage function from the language model's parametric memory and places it in a separate external knowledge base. This extraction allows the knowledge base to be independently managed, resized, or updated without modifying the language model architecture, thereby achieving memory size adaptability without retraining complexity.
2Adaptability or versatility
If the language model uses fixed parametric memory, then the model structure remains stable, but there is no control over which part of the memory stores data about what
Solution Approach 1:
The patent segments the language model into a stable core component for general language understanding and a separate, controllable knowledge base for factual information storage. This allows independent control over what data is stored where in the knowledge base without affecting the stable structure of the language model, resolving the contradiction between data storage control and structural stability.
3Reliability
If the language model memorizes knowledge during pre-training, then the model can perform knowledge intensive tasks, but the memorized knowledge becomes obsolete over time requiring re-training
Solution Approach 1:
The patent extracts the knowledge storage function from the language model and places it in a separate external knowledge base that can be independently updated. This allows the knowledge base to be refreshed with current information without requiring retraining of the language model, thus maintaining knowledge freshness while avoiding retraining requirements.
Solution Approach 2:
The patent introduces a dynamic knowledge base that can be continuously updated and modified without affecting the static language model. This dynamic structure allows the system to adapt to changing knowledge requirements and maintain up-to-date information without the complexity of retraining the entire language model.
4Reliability
If the language model uses large size to memorize more data, then the model can perform satisfactorily on knowledge intensive tasks, but the model requires very large models to memorize enough data
Solution Approach 1:
The patent segments the system into a language model for general processing and a separate knowledge base for factual information storage. This allows the knowledge base to be independently sized and optimized for storing required factual data, eliminating the need to increase the language model size to accommodate more knowledge, thus improving task performance without increasing model size.
Data Source
AI summary
The technology employs soft knowledge prompts (KPs) to inject relevant world knowledge into language models. This includes training KPs via self-supervised learning on data from one or more knowledge bases. KPs are task independent and can function as an external memory of the language models. KPs may be entity-centric, meaning that each prompt primarily encodes information about one entity from a given knowledge base. A method includes identifying a KP in response to a received input text, concatenating that KP to a sequence of word embeddings of the input text, applying the concatenated information to a trained language model, predicting an object entity name, computing a cross-entropy loss, and updating the identified KP based on the computed cross-entropy loss.