Sparse Data Interconnect for Deep Learning Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training deep learning networks has become an exascale class problem due to the need for expensive computing resources and high energy consumption, particularly because many ineffectual computations occur during the training process, which are challenging to eliminate efficiently.
Innovation Solution
The method employs sparsity by using a hardware scheduler and a low-cost sparse data interconnect to dynamically eliminate ineffectual multiply-accumulate operations during training, exploiting naturally occurring sparsity in activations and gradients without affecting convergence or accuracy, and is compatible with existing neural networks and training algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional dense computation is used for training deep learning networks, then computation accuracy is maintained, but execution time and energy consumption increase significantly
Solution Approach 1:
The patent extracts and eliminates ineffectual multiply-accumulate operations from the computation graph by identifying operations with zero operands. The hardware scheduler detects zero values in input tensors and removes corresponding MAC operations, keeping only the essential computations that contribute to the gradient update, thereby reducing both execution time and energy consumption
Solution Approach 2:
The patent applies partial action by performing only the necessary MAC operations rather than all operations in the dense computation graph. By selectively executing only non-zero operations, the system performs less computation (partial action) while achieving the same training objective, thus improving productivity without sacrificing computation accuracy
2Productivity
If sparsity is exploited to eliminate ineffectual MAC operations, then execution time and energy consumption are reduced, but system complexity increases
Solution Approach 1:
The patent introduces a hardware scheduler as an intermediary component that coordinates between the sparse data interconnect and the MAC units. This scheduler receives bit vectors indicating zero positions, determines which operations to eliminate, and manages the execution of remaining operations, thereby organizing the complexity in a structured manner rather than leaving it unmanaged
Solution Approach 2:
The patent replaces the traditional mechanical control system that executes all MAC operations with a智能化 hardware scheduler that uses bit vectors and priority schemes to dynamically determine operation execution. This substitution introduces intelligence and adaptability, managing the increased complexity through algorithmic control rather than fixed mechanical sequencing
3Loss of energy
If a hardware scheduler with bit vectors is used to identify non-zero values, then ineffectual operations are eliminated, but device complexity increases
Solution Approach 1:
The patent changes the parameter representation of input data by introducing bit vectors alongside the conventional tensor data. These bit vectors encode zero-position information in a compact binary format, allowing the system to identify and eliminate ineffectual operations without significantly increasing the data footprint or processing complexity
Solution Approach 2:
The patent creates a simplified copy of the operation schedule that excludes zero operations. The hardware scheduler generates a pruned computation schedule based on the bit vectors, copying only the necessary MAC operations into an execution queue, thereby reducing the amount of work that needs to be managed and executed while keeping the scheduling mechanism itself relatively simple
Data Source
AI summary
A system and method for using sparsity to accelerate deep learning networks. The method includes: communicating a bit vector to a scheduler identifying which values in an input tensor are non-zero; for each lane of the input tensor, determining which values are to be communicated for multiply-accumulate (MAC) operations, the determination including directing performance of one of: communicating the current value in the lane; communicating the next value in the same lane where such value is non-zero; communicating a value from a step ahead in time where such value is non-zero; and communicating a value from a neighboring lane where such value is non-zero; and outputting the values of the MAC operations.