Path-Based Neural Network Layout for Sparse Linear Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current fully-connected artificial neural networks (ANNs) are computationally expensive and time-consuming due to their quadratic nature, necessitating a need for implementations with linear complexity in time and memory space, and efficient training methods to avoid dense network training followed by compression.
Innovation Solution
Representing ANNs using paths within the network, where each path connects an input to an output via vertices and edges, and initializing weighted connections with low discrepancy sequences, allowing for sparse training and reduced complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If fully-connected neural networks are used, then comprehensive coverage of all input-output connections is achieved, but computational complexity becomes quadratic
Solution Approach 1:
The patent segments the fully-connected network into multiple sparse subnetworks, each handling a subset of connections. This segmentation maintains comprehensive coverage across all subnetworks while reducing the computational complexity of each individual subnetwork from quadratic to linear scale.
Solution Approach 2:
The patent employs partial action by training sparse subnetworks that cover only portions of the full connection space at any given time. Through multiple training passes with different subsampled subnetworks, the system achieves comprehensive coverage without requiring all connections to be active simultaneously, thus reducing computational complexity.
2Measurement precision
If dense network training is performed, then complete model accuracy is achieved, but training time and computational resources increase significantly
Solution Approach 1:
The patent implements periodic action by repeatedly training the same sparse subnetwork multiple times with different subsampled connections. This periodic retraining of limited subnetworks achieves comparable accuracy to dense training while significantly reducing training time, as each subnetwork is trained efficiently rather than training all possible connections once.
Solution Approach 2:
The patent maintains continuity of useful action by ensuring that sparse subnetworks are trained iteratively and overlapping subnetworks are used across different training phases. This continuous training approach ensures that all connections are eventually covered and refined, maintaining model accuracy while keeping the training process ongoing and efficient.
3Quantity of substance
If network compression is applied after training, then model size is reduced, but the initial training cost remains high
Solution Approach 1:
The patent applies preliminary action by pre-defining sparse subnetwork structures and connection patterns before training begins. This preliminary structuring allows the model to be trained in a sparse format from the outset, achieving both size reduction and training efficiency simultaneously, rather than training a dense network and then compressing it afterward.
4Productivity
If linear complexity implementation is used, then training speed improves, but hardware requirements and processing architecture become more complex
Solution Approach 1:
The patent segments the computational workload into independent sparse subnetworks that can be processed in parallel. This segmentation enables linear complexity implementation where each subnetwork is trained independently, improving training speed while using standard hardware architectures without requiring complex specialized hardware.
Data Source
AI summary
Artificial neural networks (ANNs) are computing systems that imitate a human brain by learning to perform tasks by considering examples. These ANNs are typically created by connecting several layers of neural units using connections, where each neural unit is connected to every other neural unit either directly or indirectly to create fully connected layers within the ANN. However, by representing an artificial neural network utilizing paths from an input of the ANN to an output of the ANN, a complexity of the ANN may be reduced, and the ANN may be trained and implemented in a much faster manner when compared to fully connected layers within the ANN. More specifically, the ANN may be trained sparse from scratch in order to avoid a more expensive procedure of training the ANN and compressing it afterwards.