Knowledge-in-Context Model for Reducing Language Model Parameters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Fully-parametric language models require a large number of parameters to store knowledge for NLP tasks and struggle to adapt to evolving world knowledge without expensive re-training, while semi-parametric models face challenges in locating and retrieving relevant knowledge pieces from sparse textual corpora.
Innovation Solution
The Knowledge-in-Context (KiC) model uses a knowledge-rich external memory with six broad knowledge categories, employing a knowledge selector to adaptively retrieve relevant knowledge pieces and augment inputs for a text-to-text model, effectively reducing the need for extensive parameterization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If fully-parametric language models are used to store knowledge for NLP tasks, then task performance is improved, but the number of model parameters increases significantly
Solution Approach 1:
The patent extracts knowledge from the parametric model and stores it in an external memory system. The parametric model only stores task-specific parameters while knowledge is stored separately in external memory, allowing the model to perform tasks without requiring large numbers of parameters to store general knowledge.
Solution Approach 2:
The patent introduces an external memory system as an intermediary between the parametric model and knowledge storage. This external memory acts as a mediator that stores knowledge and allows the parametric model to access it during task execution, reducing the burden on the model parameters themselves.
2Reliability
If fully-parametric language models are used to solve NLP tasks, then task capability is improved, but the model becomes harder to adapt to evolving knowledge without expensive re-training
Solution Approach 1:
The patent extracts knowledge from the model parameters and places it in external memory. This separation allows the parametric model to maintain its core language processing capabilities while adapting to new knowledge by simply updating the external memory, avoiding expensive re-training of the entire model.
Solution Approach 2:
The patent creates a dynamic system where knowledge can be easily updated in external memory without changing the underlying parametric model. This dynamic architecture allows the system to adapt to evolving world knowledge by simply adding or updating entries in external memory, maintaining adaptability without requiring model re-training.
3Quantity of substance
If semi-parametric models are used with external memory, then the number of parameters is reduced, but it becomes difficult to locate and retrieve relevant knowledge pieces from sparse textual corpora
Solution Approach 1:
The patent introduces a retriever module as an intermediary between the query and the external memory. This retriever uses dense vector representations and similarity search to efficiently locate and retrieve relevant knowledge pieces from the external memory, solving the difficulty of manual search through sparse textual corpora.
Solution Approach 2:
The patent replaces manual or mechanical search methods with automated vector-based retrieval mechanisms. By converting text queries and knowledge pieces into dense vector representations and using mathematical similarity calculations, the system automatically and efficiently retrieves relevant knowledge without requiring manual search through large corpora.
4Quantity of substance
If oversized text chunks are used to build indexing, then knowledge coverage is improved, but it becomes harder to determine whether knowledge is contained
Solution Approach 1:
The patent segments the external memory into smaller, more manageable text chunks rather than using oversized chunks. This segmentation allows for more precise matching and retrieval of knowledge pieces, making it easier to determine whether specific knowledge is contained in the external memory while maintaining comprehensive coverage through the organized structure.
Data Source
AI summary
A method including: receiving an input comprising natural language texts; selecting, via a knowledge selector, one of a plurality of knowledge categories from an external memory based on a context of the input; retrieving one or more helpful knowledge pieces from the selected knowledge category; augmenting the input using the one or more helpful knowledge pieces; feeding the augmented input into a text-to-text model; and generating an output answer based on the text-to-text model.