Dynamic Neural Network Generation via Sub-Network Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network architectures for edge devices require time-consuming fine-tuning and reloading/retraining when switching between sub-networks, especially due to temperature fluctuations that demand adjustments in network efficiency.
Innovation Solution
A dynamic neural network is generated by utilizing a neural architecture search (NAS) method to obtain multiple sub-networks, combining them into a combined neural network, and fine-tuning this combined network to achieve efficient and adaptable performance without the need for reloading or retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple sub-networks are deployed on an edge device and need to be fine-tuned independently to reach reasonable quality, then each sub-network can achieve good performance, but the fine-tuning process becomes time-consuming
Solution Approach 1:
The patent combines multiple sub-networks into a single unified neural network architecture that can dynamically switch between different sub-network configurations. This merging eliminates the need for independent fine-tuning of each sub-network, as they share common parameters and are trained together as one integrated system, significantly reducing the overall fine-tuning time while maintaining performance quality.
Solution Approach 2:
The unified neural network is designed with multi-functionality to perform both inference and dynamic switching between sub-networks. This universal architecture allows the system to adapt to different computational requirements and temperature conditions without requiring separate fine-tuned models for each scenario, thereby reducing time consumption while ensuring reliable performance across various conditions.
2Adaptability or versatility
If sub-networks are switched at runtime of the edge device according to working temperature, then the device can adapt to temperature changes, but switching between sub-networks is time-consuming
Solution Approach 1:
The patent implements a dynamic neural network where the switching between sub-networks is achieved through dynamic routing mechanisms rather than static model switching. The system can dynamically adjust which sub-network paths are activated based on real-time temperature conditions, allowing for rapid adaptation without the time-consuming process of reloading or reinitializing different models during runtime.
Solution Approach 2:
The sub-networks are nested within a unified neural network architecture, where smaller sub-networks are embedded within a larger framework. This nesting allows for seamless transitions between different network configurations by simply activating or deactivating specific pathways within the unified structure, eliminating the need for external model loading and reducing switching time while maintaining temperature adaptability.
3Reliability
If a supernet is pre-trained and sub-networks are obtained through NAS, then better performance sub-networks can be obtained, but the sub-networks need to be fine-tuned independently which is time-consuming
Solution Approach 1:
The patent merges multiple NAS-generated sub-networks into a single unified neural network that inherits the performance benefits of NAS while eliminating the need for independent fine-tuning. The unified network is trained once using the NAS framework, and all sub-network configurations benefit from this single training process, significantly improving fine-tuning efficiency while maintaining the high performance quality achieved through NAS.
Solution Approach 2:
The unified neural network is pre-trained with a comprehensive training process that prepares all sub-network configurations in advance. This preliminary action ensures that when the system needs to switch between sub-networks at runtime, they are already optimized and ready to use without requiring additional independent fine-tuning, thereby improving productivity while maintaining the performance benefits of NAS.
Data Source
AI summary
A method for generating a dynamic neural network includes: utilizing a neural architecture search (NAS) method to obtain a searched result, wherein the searched result comprises a plurality of sub-networks; combining the plurality of sub-networks to generate a combined neural network; and fine-tuning the combined neural network to generate the dynamic neural network.