Router-Guided Neural Network Optimization for Faster Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for accelerating neural network (NN) execution, such as using GPUs or compressing models, compromise accuracy or are computationally intensive, and Neural Architecture Search (NAS) is excessively resource-demanding.
Innovation Solution
A method to optimize a pretrained NN model by constructing a master NN model with subnetworks and a router NN to direct data instances to specific subnetworks, using a confined search space to improve performance metrics like latency and power consumption while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If Neural Architecture Search (NAS) is used to find optimal NN architecture, then execution speed and efficiency are improved, but computational resource consumption becomes excessively high (thousands of GPU hours)
Solution Approach 1:
The patent applies preliminary action by pre-defining a confined search space of neural network architectures before the search process begins. This search space is constructed based on performance requirements and hardware constraints, allowing the NAS process to focus only on relevant architectures rather than exploring the entire architecture space, thereby reducing computational resource consumption while still finding optimal solutions
Solution Approach 2:
The patent implements local quality by creating a customized search space tailored to specific performance requirements and hardware constraints. Instead of using a generic comprehensive search space, the method locally adapts the search space to the specific application context, including only architectures that meet predefined performance thresholds and hardware compatibility requirements
2Speed
If model compression techniques (pruning, quantization) are applied to accelerate inference, then execution speed is improved, but model accuracy is compromised
Solution Approach 1:
The patent applies dynamics by making the neural network architecture adaptive and configurable based on performance requirements. The system dynamically selects and configures architecture parameters (such as number of layers, filter sizes, connectivity patterns) to achieve the desired balance between inference speed and accuracy, rather than using fixed compression techniques that sacrifice accuracy
3Speed
If powerful GPUs with many computing cores are used to execute NN algorithms, then execution speed is improved, but device complexity and power consumption increase
Solution Approach 1:
The patent implements parameter changes by optimizing neural network architecture parameters (layer depths, filter counts, kernel sizes, activation functions) to match the capabilities of available hardware. The system adjusts these parameters to achieve high execution speed on the target hardware without requiring excessively complex GPU configurations, thereby reducing hardware complexity requirements
Data Source
AI summary
A system and method of automated optimization of a Neural Network (NN) model by at least one processor may include: receiving a pretrained NN model; constructing at least one master NN model, based on the pretrained NN model, each master NN model comprising a plurality of subnetworks; for each master NN model, constructing a router NN, adapted to direct one or more data instances of an input dataset to specific subnetworks of the master NN model; for each master NN model, calculating a utility score; and selecting a master NN model of the at least one constructed master NN models, based on the utility score.