Multi-Dimensional Neural Network Parameter Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face challenges in training efficiency due to the vanishing gradient problem and increased number of parameters when adding multiple hidden layers, which slows down the training process and affects accuracy.
Innovation Solution
A multi-dimensional neural network architecture is introduced, allowing connections among layers regardless of their position, with inner and outer DNNs sharing parameters and using hard or soft connections to combine intermediate outputs, preventing the formation of cycles and reducing the number of parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of hidden layers is increased to improve approximation accuracy, then the accuracy of processing input data is improved, but the training process becomes slower and the number of parameters increases
Solution Approach 1:
The patent merges multiple DNNs into a multi-dimensional neural network where corresponding layers share parameters. This combining approach allows the network to achieve the approximation capabilities of multiple deep layers while reducing the total number of parameters and training time, as the shared parameters are trained once and reused across multiple DNNs.
Solution Approach 2:
The patent implements parameter sharing across corresponding layers of multiple DNNs, where a single set of parameters serves multiple functions across different DNNs. This universal parameter usage allows the system to maintain high approximation accuracy through multiple layers while reducing the overall parameter count and training complexity.
2Measurement precision
If the number of hidden layers is increased to improve approximation accuracy, then the accuracy of processing input data is improved, but the number of parameters increases
Solution Approach 1:
The patent merges multiple DNNs into a multi-dimensional neural network where corresponding layers share parameters. This combining approach allows the network to achieve the approximation capabilities of multiple deep layers while reducing the total number of parameters, as the same parameters are reused across multiple DNNs rather than maintaining separate parameters for each.
Solution Approach 2:
The patent implements parameter sharing across corresponding layers of multiple DNNs, where a single set of parameters serves multiple functions across different DNNs. This universal parameter usage allows the system to maintain high approximation accuracy through multiple layers while reducing the overall parameter count.
3Reliability
If skip connections are added to overcome vanishing gradient problem, then the training stability is improved, but the device complexity increases
Solution Approach 1:
The patent segments the multi-dimensional neural network into multiple DNNs with skip connections between corresponding layers. This segmentation allows gradient flow through multiple paths while maintaining manageable complexity within each segment, as each DNN maintains its own structured architecture with added skip connection pathways.
Solution Approach 2:
The patent introduces skip connections as intermediary pathways that directly connect corresponding layers across different DNNs. These intermediary connections facilitate gradient flow and information exchange between layers without requiring complex reconfiguration of the base DNN architecture, thus improving training stability with moderate complexity increase.
4Productivity
If data propagation is restricted to forward direction to prevent cycles, then the computational efficiency is improved, but the information exchange capability is limited
Solution Approach 1:
The patent segments the network into multiple DNNs arranged in a multi-dimensional structure, where each DNN processes data in the forward direction. This segmentation allows information exchange between corresponding layers of different DNNs through skip connections while maintaining computational efficiency within each segment, effectively combining unidirectional processing with multi-directional information flow.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
An artificial intelligence (AI) system is disclosed. The AI system comprises an input interface to accept input data; a memory storing a multi-dimensional neural network having a sequence of deep neural networks (DNNs) with an inner DNN and an outer DNN; a processor configured to submit the input data to the multi-dimensional neural network to produce an output of the outer DNN; and an output interface to render at least a function of the output. Each DNN processes the input data sequentially by a sequence of layers along a first dimension of data propagation. The DNNs are arranged along a second dimension of data propagation from the inner DNN to the outer DNN. Further, the DNNs are connected such that an output of at least one layer of a DNN is combined with an input to at least one layer of subsequent DNN in the sequence of DNNs.