Path-Based Neural Network Representation for Linear Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current artificial neural networks (ANNs) are computationally expensive and time-consuming due to their quadratic complexity, requiring efficient linear algorithms for implementation and training.
Innovation Solution
Representing ANNs using paths within the network, where each path connects input to output with vertices representing neural units and edges as weighted connections, allowing for linear complexity by sampling paths proportionally to weights and activations, and using parallel processing units for training and inference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If fully-connected ANN implementations are used, then comprehensive neural network functionality is achieved, but computational complexity becomes quadratic and processing becomes expensive and time-consuming
Solution Approach 1:
The patent segments the fully-connected neural network into multiple independent paths, where each path represents a subset of connections from input to output. By processing these paths separately and in parallel, the computational complexity is reduced from quadratic O(n²) to linear O(n), while maintaining the comprehensive functionality of the original fully-connected network through the aggregation of path-wise computations.
2Productivity
If linear algorithms are used in hardware processing units, then processing efficiency is improved, but the ability to implement traditional quadratic ANN operations is limited
Solution Approach 1:
The patent transforms the computational parameters of the neural network by representing the weight matrix as a product of two lower-rank matrices (factorization). This parameter transformation enables the implementation of quadratic ANN operations using only linear algorithms in hardware, as the factorized form can be computed through sequential linear operations rather than requiring simultaneous quadratic computations.
Data Source
AI summary
Artificial neural networks (ANNs) are computing systems that imitate a human brain by learning to perform tasks by considering examples. By representing an artificial neural network utilizing individual paths each connecting an input of the ANN to an output of the ANN, a complexity of the ANN may be reduced, and the ANN may be trained and implemented in a much faster manner when compared to an implementation using fully connected ANN graphs.