Retrieval-Augmented Language Model Pre-Training Reduces Network Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As language models accumulate knowledge, they increase in size, leading to system operation issues such as increased storage space and complexity, making it difficult to handle new tasks effectively.
Innovation Solution
A neural-network-based textual knowledge retriever is trained alongside the language model, which retrieves and ranks relevant documents from an unlabeled corpus, allowing the model to intelligently retrieve information rather than storing all relevant data implicitly, reducing network complexity and enabling handling of new tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If language models accumulate more knowledge by storing it implicitly in network parameters, then the model's knowledge capacity improves, but the network size and complexity significantly increase
Solution Approach 1:
The patent segments the knowledge storage function by introducing a separate knowledge base component that stores documents externally, while the language model only stores compact embeddings and attention mechanisms. This divides the system into a language processing module and a knowledge storage module, allowing knowledge accumulation without proportional increase in model parameters.
Solution Approach 2:
The patent introduces an intermediary mechanism (attention mechanism with query-key-value transformations) that mediates between the language model and the external knowledge base. This intermediary allows the model to access and process knowledge from the external corpus without storing all knowledge directly in its parameters, reducing model size while maintaining knowledge access capability.
2Reliability
If language models store all potentially relevant information implicitly in parameters, then the model can answer questions accurately, but storage space requirements increase significantly
Solution Approach 1:
The patent extracts the bulk of knowledge storage from the language model parameters and places it in an external knowledge base. Only essential model components (embeddings, attention weights, architecture parameters) remain in the model, dramatically reducing storage requirements while maintaining the ability to answer questions through retrieval-augmented generation.
Solution Approach 2:
The patent creates a copying mechanism where instead of storing all knowledge directly in the model, the model learns to access copies of knowledge from an external corpus through retrieval mechanisms. This allows the model to utilize external knowledge copies without duplicating them within model parameters, reducing storage footprint.
3Productivity
If language models use implicit parameter storage for knowledge, then the model handles pre-training tasks effectively, but it becomes difficult to handle new tasks
Solution Approach 1:
The patent introduces dynamic adaptability by allowing the knowledge retrieval mechanism to be configured for different tasks and domains. The attention mechanism and retrieval system can dynamically adjust to new tasks by learning from task-specific data without requiring changes to the core model architecture, enabling the model to adapt to new tasks while maintaining pre-training performance.
Solution Approach 2:
The patent creates a universal framework where the same retrieval-augmented generation system can handle multiple tasks (question answering, text generation, classification) by dynamically retrieving relevant knowledge from the external corpus. This multi-functional design allows the model to tackle new tasks without retraining, as the universal architecture can adapt to different task requirements through the knowledge retrieval component.
Data Source
AI summary
Systems and methods for pre-training and fine-tuning of neural-network-based language models are disclosed in which a neural-network-based textual knowledge retriever is trained along with the language model. In some examples, the knowledge retriever obtains documents from an unlabeled pre-training corpus, generates its own training tasks, and learns to retrieve documents relevant to those tasks. In some examples, the knowledge retriever is further refined using supervised open-QA questions. The framework of the present technology provides models that can intelligently retrieve helpful information from a large unlabeled corpus, rather than requiring all potentially relevant information to be stored implicitly in the parameters of the neural network. This framework may thus reduce the storage space and complexity of the neural network, and also enable the model to more effectively handle new tasks that may be different than those on which it was pre-trained.