Mirror Deep Neural Networks for Fast Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face challenges in training due to the vanishing gradient problem, where error gradients exponentially decrease as they pass through multiple layers, leading to slow training and high computational costs, especially when using traditional methods like unsupervised pre-training and batch normalization.
Innovation Solution
The introduction of Mirror DNNs, which regularize to a linear function and employ a mirror layer structure with two non-linear equations to maintain linearity, allowing for faster training by only deviating from a linear function as needed, reducing the number of layers required while improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional deep neural networks with multiple layers are used to model complex non-linear relationships, then the network can represent certain function classes better, but the training speed decreases significantly due to vanishing gradient problem
Solution Approach 1:
The patent segments the deep neural network into mirror pairs of layers, where each mirror layer consists of two sub-layers with symmetric weight matrices. This segmentation allows the network to maintain depth for complex function representation while the mirror structure ensures gradients flow symmetrically, preventing vanishing gradients and enabling faster training.
Solution Approach 2:
The patent introduces asymmetric regularization that temporarily breaks the symmetry during training by applying different regularization strengths to positive and negative weight deviations from the identity matrix. This asymmetric treatment helps the network escape from symmetric solutions and learn complex non-linear relationships while maintaining the overall mirror structure for stable gradient flow.
2Reliability
If unsupervised pre-training and batch normalization are applied to address vanishing gradient problem, then training can proceed with improved parameter initialization, but computational overhead increases significantly
Solution Approach 1:
The patent extracts and eliminates the need for separate pre-training and batch normalization components by integrating their benefits directly into the mirror layer structure. The mirror symmetry inherently provides stable gradient flow without requiring batch normalization, and the structured initialization replaces the need for unsupervised pre-training, significantly reducing computational overhead.
Solution Approach 2:
The patent merges the functions of multiple separate training components (unsupervised pre-training, batch normalization, and weight initialization) into a single unified mirror layer structure with symmetric weight matrices. This consolidation achieves the same training convergence benefits while eliminating the computational overhead of multiple separate processes.
3Measurement precision
If the number of layers in the neural network is increased to improve pattern recognition performance, then the network capacity increases, but the gradient vanishes exponentially causing training to slow considerably
Solution Approach 1:
The patent segments the deep network into mirror pairs of layers, where each pair maintains symmetric weight matrices. This segmentation allows the network to achieve great depth for high pattern recognition accuracy while the symmetric structure ensures that gradients flow equally in both directions through each mirror pair, preventing exponential vanishing and enabling efficient training of very deep networks.
Solution Approach 2:
The patent changes the parameter structure by constraining weight matrices in mirror layers to be symmetric (W = W^T). This parameter change fundamentally alters the gradient flow properties, ensuring that gradients do not vanish exponentially even as the number of layers increases, thus enabling fast training of deep networks with high pattern recognition capability.
Data Source
AI summary
The mirror deep neural networks (DNNs) as described herein recognize patterns in an input signal. Mirror DNNs regularize to a linear function and train very quickly. Mirror DNNs employ a neural network pattern recognizer that receives a set of features extracted from an input signal and inputs the set of features into a multi-layer neural network. The multi-layer neural network has an input layer that receives the set of features, a plurality of intermediate layers, and an output layer that generates a set of output values that are indicative of a recognized pattern exhibited in the input signal. A first and second non-linear equation pair are chosen and applied to intermediate layers of the neural network so as to make the output values that are indicative of a pattern exhibited in the input signal linear.