Stacked Memory Partitioning for Locality and Bandwidth Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current high-performance computing and graphics processing systems face memory bandwidth limitations, with a high byte-to-floating-point operation ratio, leading to increased energy consumption and latency in memory accesses.
Innovation Solution
A stacked memory system is implemented where multiple memory dies are vertically aligned with processing tiles, allowing for localized memory access and partitioning of application programs to reduce memory bandwidth requirements and energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional memory systems are used, then memory capacity is sufficient, but memory bandwidth is insufficient leading to high energy consumption and latency
Solution Approach 1:
The patent transitions from a planar memory architecture to a three-dimensional stacked memory architecture. Memory dies are stacked vertically above processing tiles, creating multiple memory levels that can be accessed simultaneously. This vertical dimension enables parallel memory access paths, dramatically increasing memory bandwidth while reducing the distance data must travel, thereby lowering energy consumption per bit transferred.
Solution Approach 2:
The memory system is segmented into multiple independent memory dies, each capable of simultaneous access. Each memory die can service independent memory requests in parallel, effectively multiplying the total memory bandwidth. This segmentation also allows the system to serve multiple processing tiles simultaneously with reduced contention, lowering overall energy consumption.
2Quantity of substance
If memory is accessed remotely, then larger memory capacity is available, but access latency increases
Solution Approach 1:
The patent implements a nested memory hierarchy where multiple memory dies are stacked within a single package above the processing tiles. This nested structure provides large total memory capacity while keeping all memory dies physically close to the processing units. The vertical stacking creates a compact hierarchy where even the farthest memory locations are accessible with minimal latency compared to conventional distributed memory systems.
Solution Approach 2:
By stacking memory dies vertically in the third dimension, the patent achieves large memory capacity within a compact footprint while maintaining short access paths. The vertical arrangement allows multiple memory levels to be accessed in parallel from processing tiles below, reducing effective access latency despite increased total capacity.
3Productivity
If memory bandwidth is increased, then processing performance improves, but energy consumption increases
Solution Approach 1:
The vertical stacking architecture enables high bandwidth through parallel access paths in the vertical dimension. Multiple memory dies can be accessed simultaneously by multiple processing tiles, achieving high aggregate bandwidth. The short vertical distance between memory and processing layers reduces the energy cost per bit transferred, allowing high performance with lower energy loss compared to horizontal expansion approaches.
Solution Approach 2:
The patent changes the physical parameter of memory organization from planar to vertical stacking. This structural parameter change fundamentally alters the energy-performance relationship by reducing transmission distance while increasing parallel access capacity. The new architecture achieves higher performance at lower energy consumption by optimizing the spatial arrangement of memory and processing elements.
Data Source
AI summary
Embodiments of the present disclosure relate to application partitioning for locality in a stacked memory system. In an embodiment, one or more memory dies are stacked on the processor die. The processor die includes multiple processing tiles and each memory die includes multiple memory tiles. Vertically aligned memory tiles are directly coupled to and comprise the local memory block for a corresponding processing tile. An application program that operates on dense multi-dimensional arrays (matrices) may partition the dense arrays into sub-arrays associated with program tiles. Each program tile is executed by a processing tile using the processing tile's local memory block to process the associated sub-array. Data associated with each sub-array is stored in a local memory block and the processing tile corresponding to the local memory block executes the program tile to process the sub-array data.