Differentiable Neural Architecture Search for Hardware-Aware Latency Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Designing efficient and accurate convolutional neural networks for resource-constrained devices like mobile devices is challenging due to a large combinatorial design space, high computational cost of training, and optimality being conditioned on factors such as input resolution and target hardware, making it difficult to explore and deploy neural networks effectively.
Innovation Solution
The implementation of a differentiable neural architecture search (DNAS) engine that uses gradient-based optimization to search for neural network architectures from a discrete combinatorial space, directly optimizing for characteristics like latency and power consumption on target devices, allowing for automated and efficient generation and deployment of neural networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the design of the underlying neural network is improved to achieve significant accuracy improvement, then the accuracy of the system is improved, but the computational complexity increases making it more difficult to deploy on resource-constrained devices
Solution Approach 1:
The patent applies parameter changes by transforming the discrete architecture search space into a continuous differentiable space, allowing gradient-based optimization to efficiently search for optimal network parameters that balance accuracy and computational complexity for resource-constrained devices
Solution Approach 2:
The patent substitutes the traditional discrete search mechanism with a continuous gradient-based optimization mechanism. By making the architecture search differentiable, the system replaces exhaustive discrete search with efficient gradient descent, significantly reducing computational complexity while maintaining accuracy optimization
2Measurement precision
If the design space for potential convolutional neural networks is explored thoroughly, then better network architectures can be found, but the cost of training in terms of time and complexity becomes prohibitive
Solution Approach 1:
The patent merges the architecture search process with the network training process into a unified differentiable framework. By combining these previously separate processes, the system efficiently explores the design space without requiring separate exhaustive training for each candidate architecture, dramatically reducing training time
Solution Approach 2:
The patent changes the parameter space from discrete architecture configurations to continuous differentiable parameters. This transformation enables the use of gradient-based optimization methods that converge much faster than traditional discrete search methods, reducing the time required to explore the design space
3Ease of manufacture
If conventional neural network design methods are used, then deployment on target hardware is straightforward, but the networks are not optimized for specific hardware characteristics like latency and power consumption
Solution Approach 1:
The patent performs preliminary hardware-aware optimization during the network design phase by incorporating hardware characteristics (latency, power consumption) into the loss function. This preliminary action ensures networks are pre-optimized for target hardware before deployment, achieving both ease of deployment and hardware adaptability
Data Source
AI summary
Computer implemented systems are described that implement a differentiable neural architecture search (DNAS) engine executing on one or more processors. The DNAS engine is configured with a stochastic super net defining a layer-wise search space having a plurality of candidate layers, each of the candidate layers specifying one or more operators for a neural network architecture. Further, the DNAS engine is configured to process training data to train weights for the operators in the stochastic super net based on a loss function representing a latency of the respective operator on a target platform, and to select a set of candidate neural network architectures from the trained stochastic super net. The DNAS engine may, for example, be configured to train the stochastic super net by traversing the layer-wise search space using gradient-based optimization of network architecture distribution.