Neural Network Training Acceleration via Bit-Vector Activation Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The training process of neural networks is often slow and computationally intensive due to the large number of operations required for backward propagation.
Innovation Solution
A method is introduced that accelerates the training process by acquiring activations and a corresponding bit-vector, selecting non-zero activations using the bit-vector, and performing backward propagation only on these selected activations and their corresponding filters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If backward propagation is performed on all activations, then training accuracy is maintained, but training time and computational resources increase significantly
Solution Approach 1:
The patent extracts and identifies only the non-zero activations from the full activation set using a bit-vector mask. By separating the non-zero activations from zero activations, the system performs backward propagation exclusively on the extracted non-zero subset, eliminating unnecessary computations on zero values while preserving training accuracy.
Solution Approach 2:
Instead of performing backward propagation on all activations (excessive action), the patent applies partial action by processing only the non-zero activations. The bit-vector enables selective processing where computations are performed on a subset of activations rather than the complete set, reducing computational burden while maintaining sufficient training effectiveness.
2Reliability
If backward propagation is performed on all activations, then complete gradient computation is achieved, but computational resources and energy consumption increase
Solution Approach 1:
The patent extracts the essential computational subset by identifying non-zero activations through bit-vector interpretation. By extracting only the activations that contribute to gradient computation (non-zero values), the system eliminates energy-wasting operations on zero activations while preserving gradient computation completeness for the relevant subset.
Solution Approach 2:
The patent changes the parameter of activation selection from 'all activations' to 'non-zero activations only'. This parameter change is enabled by the bit-vector representation, which allows the system to selectively process activations based on their non-zero status, thereby reducing energy consumption while maintaining computational reliability.
3Productivity
If bit-vector processing is implemented to select non-zero activations, then computational burden is reduced, but additional memory access and processing overhead is introduced
Solution Approach 1:
The patent introduces a bit-vector as an intermediary data structure that mediates between the full activation set and the selected non-zero activations. This intermediary enables efficient selection without requiring complex processing logic, as the bit-vector provides a simple binary mask that can be interpreted quickly to identify relevant activations.
Data Source
AI summary
A method of accelerating a training process of a neural network includes acquiring activations used in the training process and a bit-vector corresponding to the activations, selecting activations requiring an operation from among the acquired activations by using the bit-vector, and performing backward propagation using the selected activations and filters corresponding to the selected activations.