Sparse Neural Network Topology for Faster Convergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks for modeling complex physical systems are redundant, leading to slow learning processes, high data requirements, and poor convergence, making them unsuitable for real-time simulation and dynamic modeling, especially when data is limited.
Innovation Solution
A method for constructing a sparse neural network with rapid Gauss-Newton type convergence, involving topological optimization through additive and subtractive phases, to adapt the network structure to the specific task, using a Lagrange operator for efficient error estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If redundant neural networks with many connections are used, then learning process is more robust against local minima, but learning speed decreases and data requirements increase
Solution Approach 1:
The patent extracts and removes redundant connections from the neural network, keeping only the essential connections needed for learning. This is achieved through a sparse network architecture that identifies and eliminates unnecessary weights, thereby reducing complexity while maintaining learning robustness.
Solution Approach 2:
The patent applies different connection densities to different parts of the network. Instead of uniform redundancy throughout, the network has localized dense regions where connections are critical and sparse regions where connections can be eliminated, optimizing both robustness and efficiency.
2Reliability
If neural network size is increased to avoid local minima, then learning reliability improves, but convergence rate decreases
Solution Approach 1:
The patent extracts the essential functional connections needed for reliable learning while removing extraneous connections that increase convergence time. The sparse architecture identifies minimal sufficient connectivity, achieving reliability without the time penalty of large redundant networks.
Solution Approach 2:
The patent employs dynamic connection pruning during the learning process, where connections are added or removed based on their contribution to learning. This dynamic adaptation allows the network to maintain reliability while optimizing convergence speed at each training stage.
3Adaptability or versatility
If manual trial-and-error topology construction is used, then network adaptability to specific tasks improves, but device complexity and construction time increase
Solution Approach 1:
The patent enables the neural network to automatically determine its own optimal topology through self-organizing mechanisms. The network structure emerges from the data and task requirements without manual intervention, achieving task-specific adaptability while eliminating the complexity of manual design.
Solution Approach 2:
The patent pre-defines a sparse topology framework that guides subsequent learning. This preliminary structural guidance accelerates convergence and reduces the search space, allowing the network to adapt to specific tasks efficiently without exhaustive manual trial-and-error.
4Measurement precision
If massive amounts of data are used for training redundant networks, then learning accuracy improves, but data storage and processing requirements increase
Solution Approach 1:
The patent extracts the essential patterns from smaller datasets by using a sparse network architecture that focuses learning capacity on critical features. This reduces the need for massive data volumes while maintaining learning accuracy through efficient parameter utilization.
Solution Approach 2:
The patent changes the parameter configuration from dense to sparse connectivity, fundamentally altering how the network processes information. This parameter change enables effective learning with reduced data volumes by concentrating computational resources on the most informative parameters and connections.
Data Source
Figure 1~2
Figure 3
Figure 4C~4b
AI summary
The subject of the invention is a method for constructing a forward propagation neural network, a set of nodes and of connection between the nodes forming a topology organized into layers, such that each layer is defined by a set of computable nodes that can be calculated during one and the same step, and the input of a processing node of a layer can be connected to the output of a node of any one of the previous layers, the method comprising a step of initializing a neural network according to an initial topology and at least one topological optimization phase, of which each phase comprises: - at least one additive phase comprising the modification of the topology of the network by the addition of at least one node and/or a connection link between the input of a node of a layer and the output of a node of any one of the previous layers, and/or - at least one subtractive phase comprising the modification of the topology of the network by the deletion of at least one node and/or a connection link between two layers, and in which each topology modification comprises the selecting of a topology modification from among a plurality of candidate modifications, on the basis of an estimation of the variation of the error of the network between each topology modified according to a candidate modification and the previous topology.