Partitioned Large Model Inference on Resource-Constrained Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large machine learning models are often too resource-intensive for devices with constrained memory and processing power, such as personal computers and mobile devices, making them difficult to run on these platforms.
Innovation Solution
The approach involves partitioning a large model into a smaller subset of basis models that can be stored and processed on resource-constrained devices, using a model-synthesis method where outputs from multiple basis models are combined to generate predictions, and dynamically updating these models based on usage and feedback, allowing for federated training across devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a large model is used to provide comprehensive model capabilities, then model performance and versatility are improved, but memory requirements and computational cost increase
Solution Approach 1:
The patent segments the large model into a set of basis models, each representing a subset of capabilities. Instead of storing and running the complete large model, the system stores only the necessary basis models and combines them dynamically through linear combinations, reducing memory requirements while maintaining comprehensive capabilities.
Solution Approach 2:
The system dynamically determines which basis models to combine for each specific task or device based on the embedding vector generated by the lightweight model. This dynamic selection allows the system to adapt the model composition to match device capabilities and task requirements, optimizing the balance between performance and resource consumption.
2Adaptability or versatility
If a large model is deployed on resource-constrained devices, then comprehensive functionality is achieved, but processing power requirements increase
Solution Approach 1:
By dividing the large model into smaller basis models, the system reduces the computational burden on resource-constrained devices. Only the necessary basis models are activated and processed locally, while the combination is performed efficiently through linear operations, maintaining functionality without requiring full model processing power.
Solution Approach 2:
The lightweight model acts as an intermediary that generates embedding vectors to determine which basis models should be combined. This intermediary mechanism enables devices with limited processing power to access comprehensive model capabilities without needing to run the entire large model, as the selection and combination process is computationally efficient.
3Adaptability or versatility
If all basis models are stored on a device, then complete model functionality is available, but memory space consumption increases
Solution Approach 1:
The large model is segmented into multiple basis models that can be selectively stored and combined. Instead of storing the complete large model, the system stores only the necessary basis models for each device or application, significantly reducing memory space consumption while maintaining the ability to provide comprehensive functionality when needed.
Solution Approach 2:
The system stores a partial set of basis models on each device rather than the complete set. This partial action approach allows devices with limited memory to function effectively for common tasks while maintaining the option to access additional basis models when needed, avoiding the memory consumption of storing all possible models.
Data Source
AI summary
Systems and methods for partitioning a large model that has been configured to use a model-synthesis approach in which multiple basis models are combined to generate a final output. The present technology provides systems and methods for identifying a device-specific or subject-specific subset of those basis models to be used on a given device, such that it need not store the weight matrices for the entire set of basis models, and may perform inference using only the weight matrices of the identified subset of basis models. In some examples, the subset of basis models used by a given device may be updated based on actual usage and feedback. Likewise, in some examples, the model may be trained in a federated setting in which multiple devices each utilize different subsets of the basis models, and share training signals with a full copy of the model.