Dynamic ML Model Resizing With Tensor Decomposition for Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face a significant decrease in inference accuracy when their size changes during the inference process on edge devices, despite a reduction in model size.
Innovation Solution
An inference apparatus that decomposes the weight coefficients of machine learning models using tensor decomposition methods, allowing for dynamic adjustment of model size through controlled width adjustments of decomposed tensors while maintaining inference accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If the size of the machine learning model is reduced to decrease inference processing load, then power consumption and memory capacity requirements are reduced, but inference accuracy significantly decreases
Solution Approach 1:
The patent applies segmentation by decomposing the weight coefficients of the machine learning model into multiple tensors through tensor decomposition. This allows the model to be divided into manageable segments that can be dynamically adjusted in size while preserving the essential information needed for accurate inference, thereby reducing power consumption without significantly compromising inference accuracy.
Solution Approach 2:
The patent utilizes parameter changes by dynamically adjusting the number of tensors used to represent weight coefficients based on control parameters such as power consumption levels and memory capacity. This enables the model size to be flexibly modified while maintaining inference accuracy through optimized tensor representation, resolving the contradiction between energy efficiency and precision.
2Quantity of substance
If the size of the machine learning model is reduced to optimize hardware performance, then memory capacity requirements are reduced, but inference accuracy significantly decreases
Solution Approach 1:
The patent segments the weight coefficients into multiple tensors, allowing selective use of fewer tensors when memory capacity is constrained. This segmentation enables the model to adapt its memory footprint while preserving inference accuracy through efficient tensor representation, directly addressing the contradiction between memory usage and precision.
Solution Approach 2:
The patent dynamically changes the model parameters by adjusting the number and configuration of tensors based on available memory capacity. This parameter adaptation allows the system to optimize memory usage without sacrificing inference accuracy, as the tensor representation maintains essential model information even at reduced sizes.
3Adaptability or versatility
If the machine learning model size is changed during inference from the learning stage, then adaptability to different hardware conditions is improved, but inference accuracy significantly decreases
Solution Approach 1:
The patent implements dynamics by enabling the machine learning model to dynamically adjust its size and configuration during inference based on control parameters such as power consumption and memory capacity. This dynamic adaptability allows the model to optimize performance for different hardware conditions while maintaining inference accuracy through sophisticated tensor decomposition and selection mechanisms.
Solution Approach 2:
The patent utilizes parameter changes to adapt the model configuration during inference by modifying the number and structure of tensors representing weight coefficients. These parameter adjustments are made based on hardware conditions while preserving inference accuracy through optimized tensor representation, resolving the contradiction between adaptability and precision.
Data Source
AI summary
An inference apparatus according to an embodiment of the present disclosure includes a memory and a hardware processor coupled to the memory. The hardware processor is configured to: acquire at least one control parameter of second machine learning model, the second machine learning model having a size smaller than a size of a first machine learning model input to the inference apparatus; change the first machine learning model to the second machine learning model based on the at least one control parameter; and perform inference in response to input data by using the second machine learning model.