Adaptive Machine Learning Model Lightweighting With Layer Transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models, particularly neural networks, face challenges in balancing computational efficiency with inference accuracy, especially in resource-constrained devices like mobile devices, where reducing computation without significantly degrading performance is necessary.
Innovation Solution
A method and device that adaptively lighten machine learning models by identifying optimal layers for transformation into depthwise separable convolution layers, using a parameter transformation model to reduce computational load while maintaining accuracy, based on available resources and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard convolutional layers are used in machine learning models, then inference accuracy is maintained, but computational load and resource consumption increase
Solution Approach 1:
The patent segments the convolutional layer into two separate operations: depthwise convolution (spatial filtering) and pointwise convolution (channel mixing). This segmentation allows each operation to be optimized independently, reducing overall computational load while maintaining the functional capabilities of the original convolutional layer for accurate inference
Solution Approach 2:
The patent changes the parameter representation by transforming standard convolution parameters into depthwise and pointwise convolution parameters through a parameter transformation model. This parameter transformation enables the model to achieve the same inference accuracy with reduced computational complexity by operating in a different parameter space
2Use of energy by moving object
If model lightweighting is applied to reduce computation, then resource consumption decreases, but inference accuracy deteriorates
Solution Approach 1:
The patent introduces a dynamic parameter transformation model that can adaptively adjust the transformation parameters based on the specific layer characteristics and resource constraints. This dynamic approach allows the model to maintain optimal inference accuracy while achieving lightweighting, as the transformation can be tailored to preserve critical features in each layer
Solution Approach 2:
The patent employs a feedback mechanism where the parameter transformation model is trained using ground truth lightweight parameters obtained through iterative optimization. This feedback loop ensures that the transformed parameters maintain high inference accuracy by continuously refining the transformation based on performance metrics
3Manufacturing precision
If ground truth lightweight parameters are obtained through iterative optimization, then transformation accuracy improves, but training time and computational resources increase
Solution Approach 1:
The patent performs preliminary action by pre-training the parameter transformation model using ground truth lightweight parameters obtained through iterative optimization during the training phase. Once trained, the model can rapidly transform parameters without requiring iterative optimization during deployment, thus achieving high transformation accuracy while minimizing training time and resource consumption in production
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided is a device for lightening a machine learning model. An electronic device according to an example embodiment may transform original parameters of an original machine learning model according to a lightweighting level based on a parameter transformation model. The electronic device may generate a lightweight model using the transformed parameters. The electronic device may perform an inference operation using the generated lightweight model.