Token-Level Router for LLM Factuality and Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models like ChatGPT and Llama 2 suffer from a decrease in factuality due to the alignment process, which damages the knowledge stored during pretraining, making them prone to hallucinations and unable to update knowledge efficiently.
Innovation Solution
Implement a token-level router that determines whether a response is factual, routing to a pretrained model for fact-related tokens and an aligned model for others, allowing separate updates and maintenance of knowledge without affecting instruction following abilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the model undergoes alignment training to improve instruction following, then the model becomes more helpful and harmless, but the factuality of the model decreases due to alignment tax
Solution Approach 1:
The model is segmented into two distinct components: an aligned model for instruction following and a pretrained model for factual knowledge. The router divides the response generation task between these two models, routing fact-related tokens to the pretrained model and non-fact tokens to the aligned model, thereby resolving the contradiction between instruction following and factuality
Solution Approach 2:
A router model is introduced as an intermediary component that determines which model (aligned or pretrained) should generate each token. The router acts as a mediator that dynamically selects the appropriate model based on whether the current token is fact-related, allowing both models to contribute their strengths without interfering with each other
2Adaptability or versatility
If the aligned model is updated with new knowledge, then the model learns updated information, but the whole pretraining and aligning pipeline has to be executed again which is expensive and time-consuming
Solution Approach 1:
The factual knowledge component is extracted from the aligned model and stored separately in the pretrained model. This extraction allows the pretrained model to be updated independently with new factual knowledge without affecting the aligned model's instruction following capabilities, eliminating the need to re-execute the entire alignment pipeline
Solution Approach 2:
The system is made dynamic by allowing the pretrained model to be updated independently and frequently with new knowledge, while the aligned model maintains its instruction following abilities. The router dynamically adapts to use the updated pretrained model for fact-related tokens, enabling continuous knowledge updates without time-consuming re-alignment
3Quantity of substance
If multiple models are connected to increase model capacity, then the amount of information that can be stored increases, but the total number of activated parameters remains the same increasing computation costs
Solution Approach 1:
Instead of activating all parameters of a large model, the system uses partial action by selectively activating only the necessary components. The router determines which model (aligned or pretrained) and which tokens require processing, activating only the minimal necessary parameters for each specific task, thereby reducing computation costs while maintaining model capacity
Data Source
AI summary
A method and apparatus comprising computer code configured to cause a processor or processors to receive an input question to the LLM, the LLM comprising a pretrained model and an aligned model, determine whether a portion of an unfinished response to the input question is factual, and answer the input question by at least adding to the unfinished response by a token-level router routing either to the pretrained model, based on determining that the portion is factual, or routing to the aligned model, based on determining that the portion is not factual.