Hierarchical Stream Buffering for GEMM Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for general matrix multiplication in machine learning, particularly in deep neural networks, face challenges with high double data rate (DDR) memory bandwidth requirements, which hinder computational efficiency and power management in software-defined accelerators.
Innovation Solution
The implementation of hierarchical buffering and partitioning techniques for matrix multiplication, using a programmable integrated circuit (IC), where matrices are divided into blocks and loaded into buffers according to specific row and column sizes, allowing for efficient streaming and accumulation of products in a compute array, reducing DDR bandwidth demands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If matrices are loaded entirely from DDR memory into compute arrays for multiplication, then computational accuracy is maintained, but DDR memory bandwidth consumption increases significantly
Solution Approach 1:
The patent divides large matrices into smaller sub-matrices or blocks that can be loaded into on-chip buffer memory. This segmentation allows the system to process matrix multiplication in chunks, reducing the amount of data that needs to be transferred from DDR memory at any one time while maintaining computational accuracy through systematic accumulation of partial results.
Solution Approach 2:
The patent pre-loads matrix data into on-chip buffer memory before computation begins. By preparing data in advance in high-speed buffer memory, the system reduces the need for repeated DDR memory access during computation, thereby reducing overall DDR bandwidth consumption while ensuring data is ready for accurate processing.
2Quantity of substance
If hierarchical buffering and partitioning are implemented, then DDR bandwidth consumption is reduced, but device complexity increases
Solution Approach 1:
The patent implements a hierarchical buffering structure where buffers are organized in multiple levels or nested configurations. This nesting allows data to be cached at different hierarchy levels, with higher-level buffers serving as intermediaries between DDR memory and compute arrays, thereby reducing DDR bandwidth consumption through structured data management.
Solution Approach 2:
The patent introduces additional organizational dimensions to data storage and access, such as temporal caching layers or multi-level buffer hierarchies. By adding these dimensional layers between DDR memory and compute arrays, the system reduces direct DDR bandwidth requirements while managing complexity through structured organizational frameworks.
3Speed
If data is streamed continuously from DDR memory to compute arrays, then computational speed is maintained, but power consumption increases
Solution Approach 1:
The patent introduces on-chip buffer memory as an intermediary between DDR memory and compute arrays. This intermediary buffer allows data to be transferred once from DDR to buffer, then reused multiple times for computation without repeated DDR access. This maintains computational speed by keeping data readily available in fast buffer memory while significantly reducing power consumption by eliminating frequent high-energy DDR memory transactions.
Solution Approach 2:
The patent ensures continuous data availability in on-chip buffers during computation, allowing the compute arrays to operate continuously without stalling for DDR memory access. This continuity maintains high computational speed while reducing power consumption by eliminating the energy-intensive repeated DDR memory access cycles that would otherwise be required.
Data Source
AI summary
Methods and apparatus are described for partitioning and reordering block-based matrix multiplications for high-speed data streaming in general matrix multiplication (GEMM), which may be implemented by a programmable integrated circuit (IC). By preloading and hierarchically caching the blocks, examples of the present disclosure reduce the double data rate (DDR) memory intake bandwidth for software-defined GEMM accelerators.