AI Accelerators on Memory Base Dies for Low-Latency Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing memory and storage solutions struggle to meet the high throughput and low latency demands of AI workloads, leading to challenges in data movement bandwidth and storage capacity, which overwhelms data centers and related devices.
Innovation Solution
Incorporating AI accelerators on memory base dies, specifically on high-bandwidth memory (HBM) dies, to process data queries efficiently by routing functions to either memory or compute dies based on their computational requirements, utilizing a silicon interposer for connectivity, and integrating processing units for matrix multiplication and accumulation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in external memory and processed by separate compute units, then storage capacity is improved, but data movement bandwidth and access latency worsen
Solution Approach 1:
The patent combines memory storage and AI processing functions into a single integrated device. The memory base die incorporates AI accelerator logic that can directly process data stored in the memory stacks without requiring data to be transferred to separate compute units, thereby maintaining high storage capacity while eliminating data movement bottlenecks and reducing access latency.
Solution Approach 2:
The AI accelerator logic embedded in the memory base die acts as an intermediary between the memory stacks and external compute units. It can pre-process or filter data before it leaves the memory subsystem, reducing the amount of data that needs to be transferred externally and improving effective data access speed for AI workloads.
2Productivity
If more memory bandwidth is provided to meet AI demands, then throughput is improved, but power consumption increases
Solution Approach 1:
The memory system performs AI processing functions itself through the embedded AI accelerator logic in the memory base die, eliminating the need to continuously move large amounts of data to external compute units. This self-service capability allows the system to maintain high throughput for AI workloads while significantly reducing power consumption by keeping data processing local to the memory subsystem.
3Productivity
If AI processing is performed on separate compute dies, then computational capability is improved, but data movement overhead and system complexity increase
Solution Approach 1:
The patent merges memory and AI processing functions into a single integrated device. The memory base die incorporates AI accelerator logic that can directly process data stored in the memory stacks without requiring data to be transferred to separate compute units, thereby maintaining high storage capacity while eliminating data movement bottlenecks and reducing access latency.
Data Source
AI summary
Provided are systems, methods, and apparatuses for incorporating artificial intelligence (AI) accelerators on memory base dies. In one or more examples, the systems, devices, and methods include determining at least one feature of a data query; routing, based on the at least one feature of the data query, a first function of the data query to a memory base die for processing by a processing unit on the memory base die; and processing, via the processing unit, data that a memory controller on the memory base die receives from at least one of one or more memory dies stacked on top of the memory base die.