Deep Neural Network Strata for Gradient Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks with more layers are difficult to train effectively, as the vanishing gradient problem and excessive gradient growth hinder performance beyond a certain depth, making it impractical to train deeper networks.
Innovation Solution
A deep neural network architecture organized into strata, where each stratum has individual inputs and objectives, with stratum support subnetworks that include linear nodes and autoencoders, facilitating communication and feature encoding, allowing for the training of deeper networks with fewer parameters for a given level of performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep neural networks with more layers are used to solve difficult problems, then performance is improved, but training becomes increasingly difficult due to vanishing gradient problem and excessive gradient growth
Solution Approach 1:
The neural network is segmented into multiple strata, where each stratum is a self-contained module with its own input, objective, and support subnetwork. This segmentation allows each stratum to be trained independently with localized gradient flow, preventing the vanishing gradient problem that occurs in deeply nested networks while maintaining the ability to solve complex problems through composition of multiple strata.
2Adaptability or versatility
If deeper neural networks are trained to solve more complex problems, then problem-solving capability is improved, but the number of parameters increases
Solution Approach 1:
Each stratum support subnetwork contains universal components (linear subnetworks, encoding subnetworks) that can be reused across multiple strata. These support subnetworks perform multiple functions including feature encoding, communication between strata, and maintaining gradient flow, thereby reducing the total number of parameters needed compared to having separate dedicated networks for each function.
Data Source
AI summary
A deep neural network architecture comprises a stack of strata in which each stratum has its individual input and an individual objective, in addition to being activated from the system input through lower strata in the stack and receiving back propagation training from the system objective back propagated through higher strata in the stack of strata. The individual objective for a stratum may comprise an individualized target objective designed to achieve diversity among the strata. Each stratum may have a stratum support subnetwork with various specialized subnetworks. These specialized subnetworks may comprise a linear subnetwork to facilitate communication across strata and various specialized subnetworks that help encode features in a more compact way, not only to facilitate communication across strata but also to increase interpretability for human users and to facilitate communication with other machine learning systems.