Neural Network Layer Expansion for Embedded Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training of compact neural networks in embedded systems for automotive applications is challenging due to fragile training results, as they are limited by memory and computational resources, making them less robust compared to larger networks.
Innovation Solution
The method involves expanding one or more layers of a neural network by applying input information to multiple filter kernels and summing their outputs, and then contracting them back into a single kernel after training, allowing for robust training on limited resources while maintaining a compact structure suitable for embedded hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a compact neural network structure is used to fit embedded hardware memory and computational limits, then the network can be deployed on embedded systems, but the training results become fragile and less robust
Solution Approach 1:
The training process is segmented into two distinct phases: an expansion phase where the network structure is temporarily enlarged to improve training robustness, and a contraction phase where the network is compressed back to its compact form for deployment. This segmentation allows the system to benefit from both large network training advantages and small network deployment efficiency.
Solution Approach 2:
The network structure is made dynamic by allowing it to change size during the training process. The network expands to a larger configuration during training to achieve better robustness, then contracts to a compact configuration for deployment. This dynamic restructuring resolves the contradiction between training requirements and deployment constraints.
2Reliability
If a larger neural network is used to improve training robustness, then training results become more reliable, but the network cannot be deployed on embedded hardware with limited resources
Solution Approach 1:
The network is preliminarily expanded to a larger size specifically for the training phase to achieve robust training results. After training is complete, the network is contracted back to its compact form. This preliminary expansion allows the system to achieve reliable training without permanently increasing the network size for deployment.
Solution Approach 2:
The network architecture parameters (such as number of channels, filter sizes) are changed during the training process by applying expansion factors. These parameter changes allow the network to temporarily adopt a larger configuration for training, then revert to compact parameters for deployment, resolving the resource constraint issue.
3Reliability
If the network structure is expanded during training to improve robustness, then training quality increases, but additional computational resources are required
Solution Approach 1:
The network dynamically adjusts its resource consumption by expanding only during the training phase when improved robustness is needed, and contracting during deployment when resource efficiency is critical. This dynamic behavior allows the system to optimize the trade-off between training quality and resource usage at different stages.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
The invention relates to a method for training an artificial neural network (1), the neural network (1) comprising a backbone portion (2) and an head portion (3), wherein at least one layer of the neural network (1) is expanded in order to improve training results.