Column Parallel Processor-HBM Stack Layout for Low-Latency AI Memory Access

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing Chip-on-Wafer-on-Substrate (CoWoS) technologies with high-bandwidth memory (HBM) and central processing units (CPUs) or graphics processing units (GPUs) face limitations in processing efficiency, power consumption, and space utilization.

Innovation Solution

A semiconductor package assembly is developed, featuring a column parallel processor with an associative processing unit (APU) tightly coupled with an HBM stack via through-silicon vias (TSVs), along with a switch fabric for data routing and local SRAM for temporary storage, enabling massively parallel operations and efficient data management.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional CoWoS configuration with CPU/GPU and HBM is used, then high memory bandwidth is achieved, but processing efficiency and power consumption are limited

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidpower consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent merges the processing unit and HBM stack into a single integrated package, with the processing unit directly mounted on the HBM stack. This integration eliminates the need for separate memory controllers and reduces data transfer distance, thereby improving processing efficiency while reducing power consumption compared to traditional CoWoS configurations.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent transitions from a planar CoWoS layout to a three-dimensional stacked architecture, where the processing unit is vertically mounted on the HBM stack. This vertical integration reduces the physical footprint and minimizes signal path length, enhancing processing efficiency and reducing power consumption.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Ease of operation

If HBM stack controller is added to manage HBM stack, then memory communication is enabled, but device complexity increases

Engineering Contradiction:

Improvememory communication capabilityVSAvoidnumber of controllers

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The processing unit in the patent performs multiple functions including data processing and memory control. By integrating the memory controller functionality within the processing unit itself, the system eliminates the need for separate HBM stack controllers, thereby reducing device complexity while maintaining full memory communication capability.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Adaptability or versatility

If more components are integrated on interposer, then functionality is enhanced, but area utilization and scalability are reduced

Engineering Contradiction:

Improvesystem functionalityVSAvoidinterposer area

Core Design Contradiction:

Adaptability or versatilityVSArea of stationary object

Solution Approach 1:

The patent moves from a two-dimensional interposer-based layout to a three-dimensional stacked architecture. By vertically stacking the processing unit on the HBM stack, the system enhances functionality through integration while minimizing the interposer area required, thereby improving area utilization and scalability.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

4Speed

If data transfer between HBM and processing unit is optimized, then bandwidth is improved, but latency is reduced

Engineering Contradiction:

Improvedata transfer bandwidthVSAvoiddata access latency

Core Design Contradiction:

SpeedVSLoss of time

Solution Approach 1:

The patent merges the processing unit and HBM stack into a tightly integrated package with direct electrical connections. This integration enables high-bandwidth data transfer while minimizing the number of intermediate interfaces and signal path length, thereby reducing data access latency compared to traditional separate-memory-controller architectures.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS20250081474A1Associate processing tightly coupled to high bandwidth memory

Publication Date: 2025.03.06 GSI TECHNOLOGY INC

AI summary

A semiconductor package assembly includes an interposer mounted on a package substrate, a column parallel processor mounted on and electrically connected to the interposer, and a high bandwidth memory (HBM) stack mounted on the parallel processor. The parallel processor includes a memory array with rows and columns, with operations occurring in the columns. Columns of the HBM stack are electrically connected to the columns of the parallel processor. The column parallel processor includes an associative processing unit (APU), a switch fabric for managing data routing, a local SRAM for temporary storage, and a buffer for managing data flow between the HBM stack and processing elements. The assembly is configured to process large language models and perform pattern searches within large datasets stored in the HBM stack.