Joint Neural-Hardware Search Using Supernetwork Weight Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models and hardware architectures are inefficiently matched, leading to a gap in performance and difficulty in designing hardware that meets increasing computing demands, with current optimization methods being time-consuming and often resulting in suboptimal co-designs.
Innovation Solution
A joint neural architecture and hardware architecture search is performed using a multi-objective reward function to evaluate models, hardware, and mapping strategies together, leveraging weight sharing and a supernetwork, with iterative parameter identification and convergence-based stopping criteria.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models and hardware architectures are optimized separately or sequentially, then the optimization process is simpler to manage, but the resulting co-design is suboptimal and fails to achieve full performance potential
Solution Approach 1:
The patent merges model architecture search and hardware architecture search into a unified joint search framework. The controller simultaneously searches both model space and hardware space, evaluating their compatibility through a multi-objective reward function that considers quality, performance, power, and area metrics together rather than separately.
Solution Approach 2:
The system employs a universal controller that handles multiple functions: generating model architectures, generating hardware architectures, evaluating quality metrics, evaluating performance metrics, and optimizing the joint configuration. This multi-functional approach enables comprehensive co-optimization within a single framework.
2Reliability
If joint optimization of model architecture and hardware architecture is performed using multi-trial approaches, then better co-designs can be achieved, but the search process becomes too time-consuming for large scale applications
Solution Approach 1:
The system implements feedback mechanisms where the multi-objective reward function evaluates joint configurations and provides guidance back to the controller for iterative improvement. The controller uses quality metrics and performance metrics as feedback to refine both model and hardware architectures in subsequent iterations, enabling efficient convergence toward optimal co-designs.
Solution Approach 2:
The optimization process dynamically changes multiple parameters including model architecture parameters, hardware architecture parameters, and mapping parameters. The controller adjusts these parameters iteratively based on reward signals, exploring the joint search space efficiently without requiring exhaustive multi-trial evaluations of all possible configurations.
3Power
If hardware is upgraded to provide higher computing capacity, then future model performance can be improved, but the development phase for hardware is longer and opportunities from model optimizations are missed
Solution Approach 1:
The system performs preliminary optimization of model architectures alongside hardware design rather than waiting for hardware to be finalized. By conducting joint search and optimization in parallel, the methodology captures future model architecture trends and prepares optimized models in advance, so that when hardware is ready, performance can be immediately realized without delays.
4Ease of operation
If blackbox optimizations are used to navigate joint space of hardware and mappings, then the search process is simplified, but model optimizations are not performed first, causing performance gains to be missed
Solution Approach 1:
The joint optimization process is segmented into distinct but coordinated components: model architecture generation, hardware architecture generation, quality evaluation, performance evaluation, and reward computation. Each component handles a specific aspect of the optimization, making the overall complex process manageable while ensuring that model optimizations are properly integrated with hardware optimizations rather than treated as a single blackbox.
Data Source
AI summary
Aspects of the disclosure are directed to jointly searching machine learning model architectures and hardware architectures in a combined space of models, hardware, and mapping strategies. A search strategy is utilized where all models, hardware, and mappings are evaluated together at once via weight sharing and a supernetwork. A multi-objective reward function is utilized with objectives for quality, performance, power, and area.