MoE Model Routing with Block-Sparse Weights for Adaptive Knowledge Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale fully-parametric models require a huge number of parameters and struggle to adapt to evolving world knowledge without expensive re-training, leading to inefficiencies in natural language processing and image/video tasks.
Innovation Solution
A semi-parametric language model architecture, Knowledge-in-Context (KiC), which uses a knowledge-rich external memory and a mixture-of-experts (MoE) model to dynamically select and utilize knowledge, reducing the need for extensive parameterization and enabling efficient adaptation to new tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large-scale fully-parametric models are used to solve multiple tasks in zero/few-shot setting, then problem solving capability is improved, but the number of model parameters increases hugely
Solution Approach 1:
The patent segments the model parameters into task-specific and shared components. Task-specific parameters are only activated when needed for particular tasks, while shared parameters handle general processing. This segmentation allows the model to achieve high problem-solving capability on multiple tasks without requiring a huge number of parameters to be active simultaneously, directly resolving the contradiction between reliability and quantity of parameters.
Solution Approach 2:
The patent introduces dynamic parameter routing mechanisms that adaptively select and activate parameters based on the specific task at hand. Instead of using all parameters statically for every task, the system dynamically activates only the necessary parameters, enabling high problem-solving capability with a smaller effective parameter count, thus resolving the contradiction between reliability and parameter quantity.
2Reliability
If model scale is increased to achieve problem solving capability, then performance on multiple tasks is improved, but adaptation to evolving world knowledge becomes harder without expensive re-training
Solution Approach 1:
The patent segments knowledge representation into modular task-specific components that can be independently updated. This allows the model to adapt to evolving world knowledge by updating only the relevant task-specific parameters without requiring expensive full-model retraining, thus resolving the contradiction between performance and adaptability.
Solution Approach 2:
The patent enables selective parameter updates and adaptation mechanisms that allow specific parameters to be modified or retrained independently based on new knowledge requirements. This selective parameter change approach maintains high performance on multiple tasks while enabling efficient adaptation to evolving world knowledge without expensive comprehensive retraining.
3Measurement precision
If more MoE weights are used in router computation, then routing accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent applies partial action by using only a subset of MoE weights for router computation at any given time, rather than all available weights. This selective use of weights maintains sufficient routing accuracy for effective task routing while significantly reducing computational complexity, directly resolving the contradiction between measurement precision and device complexity.
Data Source
AI summary
A method and apparatus comprising computer code configured to cause a processor or processors to form one or more routers of a mixture of experts (MoE) model by computing router parameters based on a plurality of MoE weights, implement at least one of the routers of the MoE model to compute one or more outputs by using a number of the MoE weights less than that of the plurality of MoE weights, and derive an MoE expert based on iteratively updating the router parameters according to the one or more outputs.