Dynamic Deep Neural Network Training with Adaptive Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Dynamic deep neural networks face challenges in optimizing filter usage for varying input data, leading to inefficient computation and accuracy issues, as existing methods do not effectively select or synthesize filter sets suitable for each input class during training.
Innovation Solution
A method and system for training dynamic deep neural networks that incorporate a routing module to determine the importance of filters, allowing different filter sets to be selected for each input class, by calculating a third loss that combines class determination losses from the last layer and routing module outputs, and updating weights accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If channel gating or channel mixing methods are used to reduce computation cost, then calculation cost is reduced, but filter selection accuracy and network adaptability deteriorate
Solution Approach 1:
The patent implements dynamic filter selection by training routing modules that can adaptively choose different filter sets based on input data characteristics. The routing modules dynamically adjust which filters are activated during inference, allowing the network to adapt to different input types and optimize computation for each specific case rather than using a static filter set.
Solution Approach 2:
The routing modules are trained in advance using multiple loss functions (first loss from last layer outputs, second loss from routing module outputs, and third combined loss) to learn optimal filter selection patterns. This preliminary training enables the routing modules to make accurate filter selection decisions during inference without requiring real-time complex computations.
2Device complexity
If the same filter is used for all input data, then network simplicity is maintained, but computation efficiency and accuracy for specific input classes deteriorate
Solution Approach 1:
The patent segments the filter set into multiple distinct filter sets, each specialized for specific input classes or features. The routing modules then determine which segmented filter sets to activate based on the input, replacing the monolithic single-filter approach with a modular segmented structure that balances simplicity and efficiency.
Solution Approach 2:
Different filter sets are designed with specialized local qualities tailored to specific input classes. For example, certain filters are optimized for detecting dog breeds while others are optimized for car recognition. This local quality specialization allows each filter set to perform optimally for its target class while the overall network maintains simplicity through automated routing.
3Measurement precision
If more filters are trained for better classification, then accuracy improves, but computation cost and training complexity increase
Solution Approach 1:
The routing modules are trained in advance using a combination of loss functions (first loss from last layer predictions, second loss from routing outputs, and third combined loss) to learn optimal filter selection patterns. This preliminary action enables the system to achieve high accuracy with fewer active filters during inference, as the routing modules already know which filters to activate for each input type.
Solution Approach 2:
The patent trains routing modules to select only the necessary subset of filters for each input class rather than activating all trained filters. This partial action approach maintains high classification accuracy by using only relevant filters while reducing computation cost compared to using the full filter set, effectively applying the principle of using just enough resources needed.
Data Source
AI summary
Provided are a method and system for training a dynamic deep neural network. The method for training a dynamic deep neural network includes receiving an output of a last layer of the deep neural network and outputting a first loss, receiving an output of a routing module according to an input class of the deep neural network and outputting a second loss, calculating a third loss based on the first loss and the second loss, and updating a weight of the deep neural network by using the third loss.