Disaggregated Systolic Arrays for Sparse Matrix Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graphics processing units (GPUs) face challenges in efficiently accelerating matrix operations, particularly for sparse data, due to limitations in cache access latency and the need for optimized processing techniques to enhance performance in machine learning and graphics applications.
Innovation Solution
The implementation of a systolic array architecture within a GPU, which includes disaggregated systolic processing units and packed data compression/expansion operations, exploits block sparsity and decompression information to optimize training and inference processes, thereby accelerating sparse matrix multiply operations and improving cache access efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional cache access methods are used in GPUs, then memory access is simplified, but cache access latency increases and performance deteriorates for matrix operations
Solution Approach 1:
The cache is segmented into multiple ways (e.g., 8-way set associative cache) where each way maintains separate data copies. This segmentation allows parallel access to different cache ways, reducing effective access latency by enabling simultaneous data retrieval from multiple segments rather than sequential access.
Solution Approach 2:
The patent introduces a new dimension of parallelism by accessing multiple cache ways simultaneously rather than accessing a single cache line sequentially. This transforms the cache access from a one-dimensional sequential operation to a multi-dimensional parallel operation, effectively reducing latency for matrix operation data access.
2Productivity
If dense data processing is used, then processing simplicity is maintained, but performance on sparse data deteriorates due to wasted computational resources
Solution Approach 1:
The systolic array processing elements are configured to detect and respond to local sparsity patterns in the data. Each processing element can identify zero or invalid elements in its local computation domain and skip unnecessary operations, allowing the system to adapt its processing behavior to the local data characteristics rather than uniformly processing all elements.
Solution Approach 2:
The patent implements partial action by performing computations only on non-zero elements of sparse matrices. The systolic array selectively activates processing elements based on data sparsity, executing computations only where needed rather than performing full matrix operations, thereby eliminating wasted computational resources on zero elements.
3Adaptability or versatility
If fixed function computational units are used, then hardware simplicity is maintained, but operational versatility deteriorates for supporting diverse machine learning operations
Solution Approach 1:
The systolic array processing elements are designed as universal computational units that can perform multiple operation types including matrix multiplication, convolution, and other neural network operations. Each processing element contains configurable arithmetic logic that can be programmed to execute different mathematical functions, allowing a single hardware architecture to support diverse machine learning workloads without requiring specialized fixed-function units for each operation type.
Data Source
AI summary
Embodiments described herein include software, firmware, and hardware logic that provides techniques to perform arithmetic on sparse data via a systolic processing unit. One embodiment provides techniques to optimize training and inference on a systolic array when using sparse data. One embodiment provides techniques to use decompression information when performing sparse compute operations. One embodiment enables the disaggregation of special function compute arrays via a shared reg file. One embodiment enables packed data compress and expand operations on a GPGPU. One embodiment provides techniques to exploit block sparsity within the cache hierarchy of a GPGPU.