High-Bandwidth Memory Sparse-Write Filtering for Lower Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-bandwidth memory (HBM) systems face challenges in managing data sparsity and similarity characteristics in machine-learning applications, leading to increased look-up latency and degraded bandwidth due to the difficulty in tracking zero-value locations and data locality, which are common in deep-learning models.
Innovation Solution
Implementing a sparse-dense control engine in the HBM logic die to detect and store zero-value locations and data similarity, allowing for the elimination of sparse writes and compression of similar data, thereby optimizing bus operations and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If HBM processes all data write requests including sparse data, then data completeness is maintained, but bus bandwidth is degraded and latency increases
Solution Approach 1:
The patent extracts and eliminates sparse data write requests from the HBM bus traffic by detecting zero-value locations and data sparsity characteristics before data reaches the memory. The controller identifies sparse writes and prevents them from consuming bus bandwidth, thereby improving overall memory throughput and reducing latency for non-sparse data operations.
2Productivity
If HBM tracks all data locations for sparsity detection, then data sparsity handling improves, but device complexity increases
Solution Approach 1:
The control engine segments the data processing function into distinct modules: sparsity detection unit, zero-value location tracking unit, and write request filtering unit. This segmentation allows each component to handle specific aspects of sparse data management independently, reducing overall system complexity while maintaining effective sparsity handling capabilities.
3Loss of energy
If HBM performs dense data writes, then data完整性 is maintained, but toggle-rate dynamic power increases
Solution Approach 1:
Instead of performing complete data writes for all data requests, the system applies partial writes only to non-sparse data. By detecting sparsity and selectively eliminating zero-value writes, the system reduces the volume of data transmitted over the bus, thereby lowering toggle-rate dynamic power consumption while maintaining data integrity for the subset of data that requires writing.
Data Source
AI summary
A high-bandwidth memory (HBM) includes a memory and a controller. The controller receives a data write request from a processor external to the HBM and the controller stores an entry in the memory indicating at least one address of data of the data write request and generates an indication that a data bus is available for an operation during a cycle time of the data write request based on the data write request comprising sparse data or data-value similarity. Sparse data includes a predetermined percentage of data values equal to zero, and data-value similarity includes a predetermined amount of spatial value locality of the data values. The predetermined percentage of data values equal to zero of sparse data and the predetermined amount of spatial value locality of the special-value pattern are both based on a predetermined data granularity.