Input-Adaptive DNNs with Early Exits for Mobile Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mobile video stream analytics systems face challenges in reducing computational consumption of Deep Neural Networks (DNNs) due to their fixed model sizes, which are not adaptive to the varying difficulty levels of video frames, leading to inefficiencies and high energy consumption.
Innovation Solution
The proposed system generates and implements Deep Neural Networks with flexible sizes using smart and dynamic early exit strategies, allowing for adaptive processing of video data in mobile devices by reducing memory and power consumption and limiting computational latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large high-capacity DNN model is used for all video frames, then recognition accuracy for hard-to-recognize frames is improved, but computational consumption and energy usage increase for all frames including easy ones
Solution Approach 1:
The system dynamically adjusts the DNN model capacity at runtime based on the difficulty level of each video frame. Easy frames are processed by small low-capacity models while hard frames are processed by large high-capacity models, making the system adaptive rather than static. This resolves the contradiction by allowing the system to have high reliability when needed while minimizing energy consumption when possible.
Solution Approach 2:
The system changes the parameter of model capacity dynamically based on frame difficulty characteristics. By adjusting the model size parameter according to the actual processing needs of each frame, the system achieves both high accuracy for difficult frames and low computational consumption for easy frames, resolving the trade-off between reliability and energy usage.
2Use of energy by moving object
If a small low-capacity DNN model is used for all video frames, then computational consumption is reduced, but recognition accuracy deteriorates for hard-to-recognize frames
Solution Approach 1:
The system uses dynamic model selection where the DNN capacity is adjusted based on the actual difficulty of each frame. For easy frames, small models are used to minimize energy consumption, while for hard frames, the system automatically switches to large high-capacity models to ensure recognition accuracy. This dynamic adaptation resolves the contradiction between low energy usage and high reliability.
3Adaptability or versatility
If multiple model variants with various capacities are installed to adapt to different difficulty levels, then adaptability is improved, but memory footprint increases
Solution Approach 1:
The system merges multiple model variants into a single integrated DNN model with early exit mechanisms. Instead of storing separate model files for each capacity level, the system combines them into one model that can dynamically adjust its processing depth, reducing memory footprint while maintaining adaptability to different frame difficulties.
Solution Approach 2:
The DNN model is designed to be universal and multi-functional, capable of processing both easy and hard frames within a single model structure. The model can adapt its effective capacity by early exiting at different layers, eliminating the need for multiple specialized model variants and thereby reducing memory requirements while preserving adaptability.
4Adaptability or versatility
If multiple model variants are incorporated with substantial content dynamics, then adaptability is improved, but overhead of searching for optimal model and switching models at runtime increases computational cost
Solution Approach 1:
The system merges the model selection and switching complexity into a single unified DNN model with early exit capabilities. This eliminates the need for separate model loading and switching operations, reducing runtime overhead while maintaining the ability to adapt to different frame difficulties through internal early exit mechanisms.
Solution Approach 2:
The system extracts the model capacity selection logic from the model loading process itself. Instead of loading different model variants and switching between them, the capacity selection is embedded within the single model's architecture through early exit layers, removing the overhead of model switching while preserving adaptability.
5Use of energy by moving object
If early exits are inserted at convolutional layers to reduce computation for easy frames, then computational consumption is reduced, but all filters within each layer must be processed before early exit decision
Solution Approach 1:
The system extracts only the necessary subset of filters from each convolutional layer for early exit processing, rather than requiring all filters to be processed. This selective filter extraction enables early exits to actually reduce computational consumption for easy frames while maintaining processing efficiency by avoiding unnecessary filter computations.
Data Source
AI summary
The present disclosure describes systems and methods for implementing input-adaptive neural networks. Several approaches and systems are presented for creating flexible and computationally-efficient deep neural networks that adapt their complexity based upon input data such as video streams. The approach may use fine-grained design and an automatic approach for generating optimal network architectures based on early-exit mechanisms. A neural network modified according to such principles can be implemented in various devices including mobile vision systems such as smartphones, drones, and head-mounted augmented reality devices.