Vertically Stacked HBM With Shared TSV Bus for AI Memory Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional semiconductor memory systems face limitations in bandwidth and storage capacity, particularly in high-bandwidth memory devices (HBM) due to the integration of volatile memory with external non-volatile storage, leading to bottlenecks in data transfer and power management, which is inefficient for data-intensive operations like AI/ML processing.
Innovation Solution
A vertically integrated computing and memory system that combines volatile and non-volatile memory dies within a semiconductor package, utilizing through-silicon vias (TSVs) for high-bandwidth communication, eliminating the need for interposer dies and interface components, and enabling direct stacking of memory layers on a host device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If volatile memory is integrated with external non-volatile storage, then storage capacity is increased, but data transfer bandwidth is limited due to external connection bottlenecks
Solution Approach 1:
The patent merges volatile memory and non-volatile storage into a single integrated memory device, where both memory types are coupled to the same host device through a shared high-bandwidth communication path. This eliminates the need for separate external connections and allows both memory types to operate at high speeds simultaneously, resolving the bandwidth limitation imposed by external connections.
2Device complexity
If memory systems use conventional external storage connections, then device complexity is reduced, but power consumption increases during data transfer
Solution Approach 1:
The patent transitions from conventional external storage connections to a vertically stacked three-dimensional memory architecture. By stacking volatile and non-volatile memory layers vertically and connecting them through TSVs to a shared communication path, the system achieves high-speed data transfer with reduced power consumption while maintaining manageable device complexity through standardized packaging.
3Productivity
If data-intensive operations like AI/ML processing are performed, then computational capability is improved, but access time to external storage becomes a bottleneck
Solution Approach 1:
The patent implements a unified memory architecture where both volatile and non-volatile memory are pre-configured with high-bandwidth access paths to the host device through shared communication channels. This preliminary setup eliminates access time bottlenecks during AI/ML operations by ensuring that both memory types can be accessed rapidly without requiring sequential external connections or data movement through intermediate interfaces.
Data Source
AI summary
System-in-packages (SiPs) having vertically integrated processing units and combined high-bandwidth memory (HBM) devices, and associated devices and methods, are disclosed herein. In some embodiments, the SiP includes a processing unit and a HBM device carried by the processing unit. Further, the combined HBM device can include one or more volatile memory dies and one or more non-volatile memory dies. The SiP can also include a shared through silicon via (TSV) bus that electrically couples combined HBM device can also include a shared bus that is electrically coupled to each of the processing unit, the one or more volatile memory dies, and the one or more non-volatile memory dies to establish communication paths therebetween.