AI Accelerators on Memory Base Dies for Faster Data Movement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data centers and related devices struggle to meet the increasing demands for high throughput and low latency in AI workloads due to insufficient data movement bandwidth and storage capacity, leading to challenges in processing large datasets efficiently.
Innovation Solution
Incorporating AI accelerators on memory base dies, such as high-bandwidth memory (HBM) dies, to process memory-bound functions and routing compute-bound operations to compute dies via a silicon interposer, enhancing data transfer and processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transferred between separate memory systems and processing units, then data storage capacity is maintained, but data movement bandwidth is insufficient and latency is high
Solution Approach 1:
The patent merges processing units directly onto the memory die, creating a unified structure where memory and processing coexist on the same substrate. This eliminates the need for separate data transfer pathways between memory systems and processing units, thereby increasing data movement bandwidth and reducing latency while maintaining storage capacity.
2Speed
If processing units are placed on memory base dies, then data bandwidth increases and processing speed improves, but device complexity increases
Solution Approach 1:
The patent segments the processing functionality into discrete processing units that can be independently designed and integrated onto the memory die. This modular approach allows for optimized data transfer pathways between memory and processing units while managing complexity through structured organization of components.
3Productivity
If more processing units are integrated on memory dies, then processing capability increases, but manufacturing complexity increases
Solution Approach 1:
The patent designs processing units with universal interfaces and standardized communication protocols that can be replicated across multiple instances on the memory die. This multi-functional design allows the same processing unit architecture to handle various data processing tasks, simplifying the manufacturing process while maintaining high processing capability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided are systems, methods, and apparatuses for incorporating artificial intelligence (AI) accelerators on memory base dies. In one or more examples, the systems, devices, and methods include determining at least one feature of a data query; routing, based on the at least one feature of the data query, a first function of the data query to a memory base die for processing by a processing unit on the memory base die; and processing, via the processing unit, data that a memory controller on the memory base die receives from at least one of one or more memory dies stacked on top of the memory base die.