Sparse Neural Network Training via Gradient Zeroing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face constraints in development speed and energy consumption due to the large number of operations, particularly in the back training phase, where existing techniques fail to effectively reduce the number of operations and increase speed.
Innovation Solution
A sparse training method that selectively zeros gradients of neurons based on specific conditions, such as probability, threshold, or percentage, allowing for reduced operations without affecting forward inference, using algorithms like Simulated Annealing or Genetic algorithms to adjust settings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks perform numerous operations to maintain accuracy, then model precision is improved, but energy consumption and operation time increase significantly
Solution Approach 1:
The patent changes the parameter of gradient values by selectively setting them to zero based on predefined conditions (such as gradient magnitude thresholds or random sampling). This parameter modification reduces the number of non-zero gradients that require computation during backpropagation, thereby decreasing energy consumption while maintaining model accuracy through careful selection of which gradients to zero out.
2Measurement precision
If deep neural networks perform numerous operations to maintain accuracy, then model precision is improved, but operation time increases
Solution Approach 1:
The patent modifies gradient parameters by setting selected gradients to zero, which directly reduces the computational workload during backpropagation. This parameter change leads to fewer operations required per training iteration, thereby reducing training time while preserving model accuracy through strategic gradient selection.
Solution Approach 2:
The patent applies partial action by selectively processing only a subset of gradients rather than all gradients. By identifying and zeroing out specific gradients that contribute least to model accuracy (based on thresholds or sampling), the method performs a partial computation that is sufficient to maintain accuracy while significantly reducing operation time.
3Speed
If existing techniques optimize forward inference operations, then inference speed is improved, but back training operation reduction remains ineffective
Solution Approach 1:
The patent inverts the conventional approach by applying optimization not in the forward inference direction but in the backward training direction. Instead of optimizing forward operations as traditionally done, the method selectively zeros gradients during backpropagation, thereby improving training efficiency while maintaining the benefits of forward inference optimization.
4Measurement precision
If gradient computation is performed for all neurons, then training accuracy is maintained, but computational complexity increases
Solution Approach 1:
The patent simplifies computational complexity by changing the parameter state of selected gradients from non-zero to zero. This parameter modification reduces the number of active computations required during backpropagation, thereby decreasing computational complexity while maintaining training accuracy through selective gradient preservation.
Solution Approach 2:
The patent extracts and removes specific gradients from the computation process by setting them to zero based on predefined conditions. This extraction of unnecessary gradient computations reduces overall computational complexity while maintaining training accuracy by retaining only the most important gradients for model updates.
Data Source
AI summary
A computing device, comprising: a computing module, comprising one or more computing units; and a control module, comprising a computing control unit, and used for controlling shutdown of the computing unit of the computing module according to a determining condition. Also provided is a computing method. The computing device and method have the advantages of low power consumption and high flexibility, and can be combined with the upgrading mode of software, thereby further increasing the computing speed, reducing the computing amount, and reducing the computing power consumption of an accelerator.