Sparsified Neural Network Training with TR-L-BFGS
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training of multilayer feedforward artificial neural networks is computationally intensive due to the large number of edges in the directed acyclic graph, leading to excessive resource consumption and time, as all edges contribute equally to the computational burden regardless of their weight, necessitating a method to simplify the graph without compromising functionality.
Innovation Solution
The method involves sparsification of the neural network by selectively removing edges with nearly zero weights, using a combination of edge and node tables, and metadata for efficient storage and processing, along with quasi-Newton optimization methods like TR-L-BFGS to iteratively adjust weights and improve accuracy, while maintaining the mathematical functionality of the network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all edges are retained in the neural network graph, then the network maintains full functionality and accuracy, but the computational burden and resource consumption increase significantly
Solution Approach 1:
The patent extracts and removes edges with weights below a certain threshold from the neural network graph. By identifying and eliminating these negligible-weight edges, the system reduces the computational burden while preserving the essential functionality of the network, thus resolving the contradiction between maintaining full functionality and improving training speed.
Solution Approach 2:
The patent applies different treatment to different edges based on their local properties (weight magnitudes). Edges with significant weights are retained to maintain network accuracy, while edges with negligible weights are removed to reduce computational burden. This local differentiation allows the system to optimize performance without compromising overall functionality.
2Productivity
If edge sparsification is applied to reduce computational burden, then training speed improves, but network accuracy may deteriorate
Solution Approach 1:
The patent dynamically adjusts the weight threshold parameter during the sparsification process. By changing this parameter, the system can control the degree of sparsification and balance between training speed and accuracy. The threshold is adjusted based on the distribution of edge weights and the desired level of sparsification, allowing flexible optimization of the trade-off between productivity and reliability.
3Measurement precision
If the full Hessian matrix is computed for optimization, then optimization accuracy is maximized, but memory consumption and computational complexity increase
Solution Approach 1:
The patent segments the Hessian matrix computation by focusing only on the sparse subset of edges that remain after sparsification. Instead of computing and storing the full dense Hessian matrix, the system computes a sparse Hessian matrix that contains only the relevant elements corresponding to the retained edges. This segmentation dramatically reduces memory footprint and computational complexity while maintaining optimization accuracy.
Data Source
AI summary
Techniques herein train a multilayer perceptron, sparsify edges of a graph such as the perceptron, and store edges and vertices of the graph. Each edge has weight. A computer sparsifies perceptron edges. The computer performs a forward-backward pass on the perceptron to calculate a sparse Hessian matrix. Based on that Hessian, the computer performs quasi-Newton perceptron optimization. The computer repeats this until convergence. The computer stores edges in an array and vertices in another array. Each edge has weight and input and output indices. Each vertex has input and output indices. The computer inserts each edge into an input linked list based on its weight. Each link of the input linked list has the next input index of an edge. The computer inserts each edge into an output linked list based on its weight. Each link of the output linked list comprises the next output index of an edge.