On-Device DNN Inference Optimization via Parallel Branch Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
On-device inference in deep neural networks (DNNs) faces challenges with high inference time and power consumption due to resource constraints in mobile devices, particularly when dealing with video-based and low-latency applications, where the performance of neural processing units (NPUs) drops significantly with increased configuration degrees.
Innovation Solution
The method optimizes convolution layers by combining parallel branches and modifying resampling layers based on specific conditions, including the placement of convolution layers in parallel branches, the presence of specified resampling ratios, and the sequencing of resampling and convolution layers, to reduce inference time and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard DNN architecture is used for on-device inference, then model accuracy is maintained, but inference time increases and power consumption increases
Solution Approach 1:
The patent segments the DNN architecture into multiple parallel branches with different complexity levels. Each branch processes features at different resolutions and depths, allowing the system to distribute computational load and reduce overall inference time while maintaining accuracy through multi-path feature extraction
Solution Approach 2:
The patent implements dynamic branch selection where the network adaptively activates or deactivates specific parallel branches based on input characteristics and resource availability. This dynamic adjustment optimizes inference speed for different scenarios while preserving accuracy when needed
2Measurement precision
If standard DNN architecture is used for on-device inference, then model accuracy is maintained, but power consumption increases
Solution Approach 1:
The patent divides the computational workload into segmented parallel branches, allowing power-efficient processing by activating only necessary branches for each inference task, thereby reducing overall power consumption while maintaining accuracy through selective feature processing
Solution Approach 2:
The patent changes architectural parameters by varying the number of filters, layer depths, and resolution levels across parallel branches. This parameter diversification enables the network to process information more efficiently, reducing redundant computations and lowering power consumption
3Power
If NPU processes high configuration degree operations, then computational capability is utilized, but performance drops due to resource constraints
Solution Approach 1:
The patent segments heavy computational operations into smaller, manageable parallel branches with controlled complexity. Each branch operates within resource constraints while collectively achieving high-level feature extraction, preventing performance degradation from any single overloaded operation
Solution Approach 2:
The patent transforms the computational problem by adding the dimension of parallelism through multiple branches. Instead of increasing complexity in a single sequence, the network distributes computations across parallel dimensions, utilizing NPU capabilities more effectively without overwhelming resource constraints
Data Source
AI summary
The disclosure relates to method and system for on-device inference in a deep neural network (DNN). The method comprises: determining whether one or more layers of the DNN satisfy one of a first, a second and a third condition, the one or more layers including one or more convolution layers and one or more resampling layers; performing the on-device inference based on the determination, wherein performing the on-device inference comprises at least one of: optimizing the one or more convolution layers in the one or more parallel branches based on the one or more layers of the DNN satisfying the first condition, optimizing the at least one of the resampling layers based on the one or more layers of the DNN satisfying the second condition, and modifying operation of the at least one of the resampling layers based on the one or more layers of the DNN satisfying the third condition.