Distributed ML Inference Partitioning to Limit Quantization Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying machine learning models trained on powerful processors to embedded processors is time-consuming and prone to quantization loss, affecting accuracy due to the need for transforming floating-point models to fixed-point versions, which is not straightforward across different instruction sizes.
Innovation Solution
Partition the machine learning model into a fixed-point model feature extractor and a floating-point model feature aggregator, where the extractor is deployed on the embedded processor and the aggregator on a separate processing device, reducing data transfer and maintaining accuracy by minimizing quantization loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If the machine learning model is transformed through quantization process to reduced instruction size for embedded processor deployment, then the model can be executed on embedded processors with less hardware resources, but quantization loss occurs that affects the accuracy of the model
Solution Approach 1:
The machine learning model is divided into two separate models: a fixed-point model feature extractor deployed on the embedded processor and a floating-point model feature aggregator deployed on a processing device. This segmentation allows different parts of the system to use different precision formats optimized for their specific functions, reducing overall quantization loss while maintaining energy efficiency on the embedded processor.
2Volume of moving object
If the machine learning model is transformed through quantization process to reduced instruction size, then the instruction size is reduced for embedded processor, but the transformation is time consuming and not straightforward
Solution Approach 1:
The model is segmented into fixed-point and floating-point components that are deployed separately. The fixed-point feature extractor is optimized for embedded processors with reduced instruction sizes, while the floating-point feature aggregator handles complex operations on a more powerful processing device, eliminating the need for time-consuming full-model quantization.
Solution Approach 2:
The model architecture is designed in advance with separate feature extractor and feature aggregator components, allowing the fixed-point version to be prepared separately for embedded deployment without requiring post-training quantization of the entire model, thus reducing deployment time.
3Loss of energy
If the entire machine learning model is deployed on the embedded processor, then execution can be performed locally with less data transfer, but the embedded processor lacks the hardware resources to execute the full floating-point model efficiently
Solution Approach 1:
The model is divided such that the computationally intensive floating-point feature aggregator is separated from the fixed-point feature extractor. This allows the embedded processor to handle local fixed-point operations while offloading complex floating-point computations to a remote processing device, balancing resource constraints with computational requirements.
Solution Approach 2:
The fixed-point model feature extractor acts as an intermediary between the embedded processor and the floating-point model feature aggregator. It performs preliminary processing locally and transfers only essential extracted features to the processing device, reducing data transfer requirements compared to transmitting entire input data.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Distributed machine learning inference includes acquiring, by an input device, an input frame, executing, by an embedded processor of the input device, a model feature extractor on the input frame to obtain extracted features of the input frame, and transmitting the extracted features from the input device to a processing device. The processing device executes a model feature aggregator to process the plurality of extracted features and obtain a model result.