ML Model Weight Decomposition for Resource-Constrained Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models face high resource overhead when updating, making it difficult to deploy them on resource-constrained devices like drones, robots, and smartphones, and updates often require significant communication overhead and lack customization for individual users.
Innovation Solution
Transforming the machine learning model by decomposing the weight matrix into a dictionary matrix and a coefficient matrix, allowing for reduced resource requirements and enabling updates on devices with limited resources, known as 'few-shot learning', where only a portion of the model is updated without modifying existing weights or data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a trained machine learning model is deployed on resource-constrained devices, then the model can perform cognitive tasks on portable platforms, but the resource overhead for updating the model becomes prohibitively high
Solution Approach 1:
The patent segments the machine learning model into two distinct parts: a fixed backbone network (pre-trained model) and a trainable head network (adaptable component). This segmentation allows the resource-constrained device to only train and update the small head network while keeping the large backbone network frozen, dramatically reducing the computational resources and energy required for model updates on portable platforms.
Solution Approach 2:
The patent extracts the computationally intensive feature extraction functionality into a separate pre-trained backbone network that is deployed once and never updated. This extraction isolates the heavy computational burden from the update process, allowing the head network to be trained with minimal resources on portable devices while maintaining access to powerful pre-extracted features.
2Adaptability or versatility
If a trained machine learning model is deployed on resource-constrained devices, then the model can perform cognitive tasks on portable platforms, but the communication overhead for model updates becomes significant
Solution Approach 1:
The patent segments the model update process so that only the small head network parameters need to be transmitted and updated, while the large backbone network remains fixed. This segmentation reduces the amount of data that needs to be communicated between cloud servers and portable devices, minimizing communication overhead while still enabling model customization.
3Productivity
If traditional model training is performed on resource-constrained devices, then the model can be updated with new data, but the training process requires large amounts of training data and numerous iterations
Solution Approach 1:
The patent performs preliminary action by pre-training the backbone network on large datasets in the cloud before deployment. This preliminary training extracts general features that are transferred to the portable device, where only the head network needs to be quickly adapted to specific tasks using minimal data. This eliminates the need for extensive training data and numerous iterations on resource-constrained devices.
Solution Approach 2:
The patent changes the training parameters by freezing the backbone network weights and only training the head network parameters. This parameter change dramatically reduces the number of trainable parameters, allowing for efficient few-shot learning with limited data and computational resources on portable platforms while maintaining high productivity.
Data Source
AI summary
A method may include a transforming a trained machine learning model including by replacing at least one layer of the trained machine learning model with a dictionary matrix and a coefficient matrix. The dictionary matrix and the coefficient matrix may be formed by decomposing a weight matrix associated with the at least one layer of the trained machine learning model. A product of the dictionary matrix and the coefficient matrix may form a reduced-dimension representation of the weight matrix associated with the at least one layer of the trained machine learning model. The transformed machine learning model may be deployed to a client. Related systems and computer program products are also provided.