Neural Network Architecture Conversion for Fast Knowledge Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of training large and small neural networks with large datasets is computation-intensive and time-consuming, making it difficult to adopt and apply deep learning-based object recognition systems in resource-poor automobile controllers.
Innovation Solution
A method for converting a first neural network with a first architecture into a second neural network with a different architecture by iteratively converting, randomly initializing, and fine-tuning layers, allowing knowledge transfer with smaller datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large neural networks with high depth are trained with large datasets to achieve high object recognition rates, then recognition accuracy is improved, but training time and computational resources increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-training a large source neural network on extensive datasets before converting it to the target architecture. The source network is trained in advance to achieve high recognition accuracy, and then this pre-trained knowledge is transferred through conversion to the target network, avoiding the need to train the target network from scratch and significantly reducing training time.
Solution Approach 2:
The patent employs parameter changes by systematically transforming the architecture parameters of the source neural network to match the target neural network. This includes converting layer types, adjusting filter numbers, modifying layer depths, and changing activation functions while preserving the learned parameters and knowledge from the source network, enabling the target network to achieve high accuracy without extensive retraining.
2Measurement precision
If large neural networks are trained with large datasets to achieve high object recognition rates, then recognition accuracy is improved, but computational resources and hardware requirements increase
Solution Approach 1:
The computational burden is shifted to the pre-training phase of the source network, which can be performed on powerful hardware. The converted target network then inherits this pre-trained knowledge, requiring minimal computational resources for fine-tuning on the target platform, thus resolving the contradiction between achieving high accuracy and reducing computational resource requirements.
Solution Approach 2:
The patent creates a copy of the source neural network's knowledge and parameters in the target neural network through the conversion process. The target network copies the learned features, weights, and representations from the source network, allowing it to achieve similar recognition accuracy without requiring the same extensive training resources.
3Device complexity
If the architecture of the neural network is changed to reduce depth and complexity for resource-poor controllers, then hardware resource requirements are reduced, but direct adoption and fine-tuning become difficult or impossible
Solution Approach 1:
The patent systematically changes architectural parameters including layer types, filter numbers, layer depths, and activation functions to transform the source network into the target network architecture. This structured approach to parameter transformation makes the conversion process methodical and manageable, resolving the difficulty of adapting networks across different architectures.
Solution Approach 2:
The conversion method achieves universality by being applicable to different types of neural network architectures. The systematic parameter transformation approach can handle various source and target architecture combinations, making the conversion process broadly applicable rather than architecture-specific, thus improving ease of manufacture.
4Adaptability or versatility
If neural networks are converted between different architectures to suit resource constraints, then hardware compatibility is improved, but knowledge transfer between architectures becomes challenging
Solution Approach 1:
The patent transforms architectural parameters while preserving the essential knowledge and learned representations from the source network. By systematically mapping parameters between architectures and maintaining the core learned features, the method enables effective knowledge transfer despite architectural differences, resolving the contradiction between hardware compatibility and knowledge retention.
Data Source
AI summary
The invention relates to a method for converting a first neural network with a first architecture into a second neural network with a second architecture for use in a vehicle controller in order to obtain the knowledge of the first neural network and transfer same to the second neural network. In a first step of the method, a conversion (701) of at least one layer of the first neural network into at least one layer of the second neural network is carried out. In a second step, a random initialization (702) of the at least one converted layer is carried out in the architecture of the second neural network. In a third step, a training process (703) of the at least one converted layer is carried out in the second neural network. In a fourth step, a fine-tuning process (704) of the non-converted layer is carried out in the second neural network or in the entire second neural network. The conversion of the first neural network into the second neural network is carried out in multiple cycles or iterations, wherein for each cycle, the conversion (701), random initialization (702), training (703), and simultaneous fine-tuning (704) steps are carried out.