Stochastic Neural Network Selective Node Activation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks (ANNs) face challenges in parallel computation and dynamic resource allocation due to the need to compute the entire network for every data query and training, limiting their ability to handle varying computational demands.
Innovation Solution
The implementation of stochastic neural networks (SNNs) with selectively activated nodes, allowing for parallel execution of queries and adaptive resource allocation by sampling computational paths and using activation probabilities to determine node activation, enabling efficient computation and training of large networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the entire neural network is computed for every data query and training, then the network maintains deterministic output, but the network cannot run multiple computations in parallel and cannot dedicate variable computational resources
Solution Approach 1:
The patent applies dynamics by making node activation probabilistic rather than deterministic. Each node has an activation probability that determines whether it fires, allowing the network to dynamically adapt its computational structure based on input difficulty. This enables variable computational resources to be dedicated to different queries while maintaining acceptable output consistency through the stochastic nature of the activation process.
Solution Approach 2:
The patent implements partial action by selectively activating only a subset of nodes in the network for each query rather than computing the entire network. The activation probability controls the extent of partial computation, allowing the network to dedicate variable computational resources depending on query difficulty while still producing meaningful outputs.
2Adaptability or versatility
If stochastic activation is used to enable parallel computations, then variable computational resources can be allocated, but the output becomes different across multiple evaluations with fixed input
Solution Approach 1:
The patent changes the activation parameter from a binary deterministic state to a probabilistic state with adjustable activation probability. This parameter change enables adaptive resource allocation by controlling the likelihood of node activation, allowing the network to flexibly dedicate computational resources while managing the trade-off with output determinism through the probability parameter.
3Loss of energy
If selective node activation is implemented, then computational resources are reduced and parallel computations are enabled, but the network requires new training approaches and path sampling mechanisms
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing activation probabilities for nodes during the training phase. These pre-computed probabilities are then used during inference to guide selective node activation without requiring complex real-time computations. This preliminary preparation reduces online computational resource consumption while managing the complexity of the selective activation mechanism.
Solution Approach 2:
The patent introduces activation probability as an intermediary element between the deterministic training process and the stochastic inference process. This intermediary allows the network to bridge the gap between traditional deterministic computation and selective stochastic activation, enabling parallel computations and reduced resource consumption while maintaining a connection to the original training objectives through the probability distribution.
Data Source
AI summary
Embodiments of the present disclosure implement a stochastic neural network (SNN) where a subset of the nodes in the network are selectively activated based on sampling a plurality of computational paths traversing the network and based on different minimum thresholds for activation. In various embodiments, an output of the stochastic neural network is a sequence of the sampled plurality of computational paths with a corresponding sequence of output values that represent approximations of the output of the stochastic neural network. The nodes can include at least one input node, at least one output node and at least two hidden nodes, wherein the hidden nodes are positioned between the input node and the output node, and wherein sampling the plurality of computational paths involves initiating each of the plurality of computational paths from a first of the hidden nodes, wherein the first of the hidden nodes has been activated by a previous computational path.