Path-Based Neural Network Representation for Linear Complexity

Solution Overview

Problem

Current artificial neural networks (ANNs) are computationally expensive and time-consuming due to their quadratic complexity, requiring efficient algorithms for linear complexity in time and memory space.

Innovation Solution

Representing ANNs using paths within the network, where each path connects inputs to outputs, allowing for linear complexity by sampling connections proportionally to weights and activations, and using parallel processing units for training and implementation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If fully-connected ANN implementations are used, then comprehensive connectivity and coverage are achieved, but computational complexity becomes quadratic and processing becomes time-consuming

Engineering Contradiction:

Improvenetwork connectivityVSAvoidcomputational complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments the fully-connected neural network into multiple paths, where each path represents a subset of connections from input to output. Instead of computing all possible connections simultaneously (quadratic complexity), the network processes information through divided paths, reducing the computational burden while maintaining comprehensive coverage through the collective set of paths.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If fully-connected ANN implementations are used, then comprehensive connectivity is achieved, but processing time increases

Engineering Contradiction:

Improvenetwork connectivityVSAvoidprocessing speed

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent segments the fully-connected neural network into multiple paths, where each path represents a subset of connections from input to output. Instead of computing all possible connections simultaneously (quadratic complexity), the network processes information through divided paths, reducing the computational burden while maintaining comprehensive coverage through the collective set of paths.

Inventive Principle:

Principle #1Segmentation

3Productivity

If linear complexity algorithms are implemented, then processing speed improves, but hardware efficiency is reduced due to inability to utilize parallel processing

Engineering Contradiction:

Improveprocessing speedVSAvoidhardware efficiency

Core Design Contradiction:

ProductivityVSEase of manufacture

Solution Approach 1:

The patent segments the neural network into multiple independent paths that can be processed in parallel. Each path contains a subset of the total connections, allowing hardware to process multiple paths simultaneously using parallel processing units, thereby maintaining hardware efficiency while achieving linear complexity through the segmented structure.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11972354B2Representing a neural network utilizing paths within the network to improve a performance of the neural network

Publication Date: 2024.04.30 NVIDIA CORP

AI summary

Artificial neural networks (ANNs) are computing systems that imitate a human brain by learning to perform tasks by considering examples. By representing an artificial neural network utilizing individual paths each connecting an input of the ANN to an output of the ANN, a complexity of the ANN may be reduced, and the ANN may be trained and implemented in a much faster manner when compared to an implementation using fully connected ANN graphs.