Sparse Matrix Memory Streaming with FIFO-Ordered Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computing devices face inefficiencies in handling sparse matrix calculations due to high hardware costs and poor memory management, particularly when dealing with large sparse matrices, leading to performance issues like latency spikes and inefficient use of cache structures.
Innovation Solution
A computing device optimized for sparse matrix calculations, featuring a main memory configured to store sparse matrices in dense vector formats, a streamer with an indexed loading block and FIFO request memory, and an indexed loading engine to manage memory access efficiently, using firmware to control data sequencing and storage in secondary memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional caches are used in sparse matrix computing devices, then memory access can be performed, but hardware cost becomes very high and performance is degraded due to latency spikes
Solution Approach 1:
The patent replaces expensive, complex conventional cache structures with a simple FIFO (First-In-First-Out) buffer structure that is much cheaper to implement in hardware. The FIFO buffer temporarily holds data elements while they are being processed, eliminating the need for complex cache management logic and associated hardware costs, while maintaining performance stability by preventing latency spikes through orderly data delivery.
Solution Approach 2:
The patent introduces a streamer component as an intermediary between main memory and the computing unit. This streamer includes a FIFO buffer that mediates data flow, ensuring that data is delivered in the correct order to computing units without requiring complex cache structures. The streamer acts as a buffer zone that decouples memory access from computation, eliminating latency spikes while keeping hardware simple.
2Quantity of substance
If CSR format is used to store sparse matrices, then memory storage size is decreased, but memory access efficiency is reduced due to double indirection on vector elements
Solution Approach 1:
The patent pre-organizes data in the FIFO buffer according to the processing sequence before data is actually needed by the computing unit. By anticipating the order in which data elements will be accessed and arranging them accordingly in advance, the system eliminates the need for double indirection lookups during computation, thereby maintaining both compact storage and high access efficiency.
Solution Approach 2:
The patent replaces the complex mechanical lookup process of CSR format (which requires two levels of indirection through index arrays) with a simpler sequential access mechanism using FIFO buffers. Data is loaded once into the FIFO in the correct order and then accessed sequentially, substituting the complex index-following mechanism with a simpler queue-based approach that maintains efficiency while keeping storage compact.
3Productivity
If inner-product algorithms are used for sparse matrix-vector multiplication, then computation can be performed, but cache utilization is poor due to disordered access patterns
Solution Approach 1:
The streamer with FIFO buffer acts as an intermediary that reorders data flow to match the computational requirements of inner-product algorithms. Instead of allowing disordered access patterns to directly impact cache performance, the FIFO buffer receives data in any order from memory and delivers it in the optimal sequence for computation, thereby improving cache utilization efficiency without sacrificing computation capability.
Solution Approach 2:
The patent makes the data delivery mechanism dynamic by using a FIFO buffer that can adapt to different access patterns. The buffer dynamically adjusts the order in which data is delivered to the computing unit based on the actual computation needs, transforming the static, inefficient cache access pattern into a dynamic, optimized data flow that improves energy efficiency while maintaining full computation capability.
Data Source
AI summary
Computing device (100) comprising a main memory (104) configured to store a sparse matrix in a dense vector format (106, 108, 110) and to store a second vector (112) or a second matrix, a computing unit (102) configured to multiply the sparse matrix by the second vector or by the second matrix, and a streamer (114) comprising: an indexed loading block (116) comprising a secondary memory (118) and a FIFO memory (120) for requests to send values stored in the secondary memory to the computing unit; an indexed loading engine (122) configured to sequentially generate and store requests in the FIFO request memory according to an order in which the values are intended to be sent to the computing unit; the request storage order being calculated and stored in the form of firmware (124).