Neural Network Early-Exit Pruning for Low-Power Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural network (DNN) models require significant computational resources and power due to processing all frames through all layers, despite varying information content, leading to inefficiencies in embedded systems and edge devices.
Innovation Solution
Identify saliency layers producing outputs close to the final output, determine intermediate exit gates using Neural Architecture Search (NAS), and prune layers post-identification to create an optimal neural network model with reduced computational costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all frames are processed through all layers of the neural network model, then the model maintains high accuracy, but computational resources and power consumption increase significantly
Solution Approach 1:
The neural network model is segmented into multiple layers with intermediate exit gates inserted at specific positions. This allows the model to be divided into independent processing segments, where simple frames can exit early after fewer layers while complex frames continue through all layers, thereby reducing average power consumption while maintaining accuracy for difficult cases
Solution Approach 2:
The model dynamically adjusts the processing depth for each frame based on its complexity and information content. By using confidence scores and exit gate mechanisms, the system adaptively determines how many layers to process each frame through, optimizing the balance between accuracy and energy consumption on a per-frame basis
2Productivity
If intermediate exit gates are added to enable early exiting, then computational costs are reduced for simple frames, but device complexity increases
Solution Approach 1:
Intermediate exit gates are pre-configured at specific layers during model construction, with predetermined positions optimized for early exiting. This preliminary structuring allows the model to naturally support variable-depth processing without requiring complex runtime decisions about where to exit, reducing operational complexity while enabling fast inference for simple cases
Solution Approach 2:
Exit gate mechanisms serve as intermediary components between layers, acting as decision points that mediate whether to continue processing or exit early. These intermediaries simplify the control logic by providing clear transition points, making the variable-depth processing mechanism more manageable and less complex than alternative approaches
3Use of energy by moving object
If layers are pruned to reduce model size, then computational resources are reduced, but model accuracy may deteriorate
Solution Approach 1:
Different parts of the model are assigned different functions based on frame complexity. Early layers handle simple classification tasks for easy frames, while deeper layers process complex frames requiring more analysis. This local differentiation allows pruning of redundant processing paths without sacrificing accuracy for complex cases, optimizing the trade-off between computational efficiency and model accuracy
Data Source
AI summary
A method for generating an optimal neural network (NN) model may include determining intermediate outputs of the NN model by passing an input dataset through each intermediate exit gate of the plurality of intermediate exit gates, determining an accuracy score for each intermediate exit gate of the plurality of intermediate exit gates based on a comparison of the final output of the NN model with the intermediate output, identifying an earliest intermediate exit gate that produces the intermediate output closer to the final output based on the accuracy score, and generating the optimal NN model by removing remaining layers of the plurality of layers and remaining intermediate exit gates of the plurality of intermediate exit gates located after the determined earliest intermediate exit gate.