Multi-Die Matrix Multiplication Using FeRAM for Low-Latency AI

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing AI processor systems face challenges in reducing latency and power consumption during the training and inference processes, which are hardware intensive activities.

Innovation Solution

The proposed solution involves a low latency and high bandwidth AI processor architecture that segregates matrix multiplication processes between a memory die and a compute die, utilizing ferroelectric memory for high-speed and low-power storage of weights and inputs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If matrix multiplication is performed using conventional AI processor systems, then computation can be completed, but latency is high and power consumption is excessive

Engineering Contradiction:

ImprovelatencyVSAvoidpower consumption

Core Design Contradiction:

Loss of timeVSUse of energy by moving object

Solution Approach 1:

The patent divides the matrix multiplication computation into multiple stages, separating weight storage (in Fe-RAM on memory die) from computation (on compute die). This segmentation allows parallel access to weights and inputs while performing multiplication, reducing latency and power consumption by avoiding repeated memory accesses.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces ferroelectric RAM (Fe-RAM) as an intermediary storage medium between conventional memory and compute units. Fe-RAM serves as a high-speed, low-power buffer that retains weight values without continuous power, enabling fast retrieval during inference while minimizing energy consumption compared to conventional DRAM or SRAM.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Speed

If weights are stored in conventional memory, then data can be accessed, but energy consumption is high due to continuous power requirements

Engineering Contradiction:

Improveaccess speedVSAvoidenergy consumption

Core Design Contradiction:

SpeedVSUse of energy by moving object

Solution Approach 1:

The patent changes the physical state parameters of the storage medium by using ferroelectric material instead of conventional volatile memory. Fe-RAM maintains data through remanent polarization states that persist without power, achieving both fast access speeds (comparable to SRAM) and ultra-low power consumption (approaching non-volatile memory) during weight storage and retrieval operations.

Inventive Principle:

Principle #35Parameter changes

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

This approach improves the performance of matrix multiplication, reduces energy consumption by up to 10 times, and achieves significant speedup across various matrix sizes, especially for larger dimensions.

Implementation Method 1

utilizing ferroelectric memory for high-speed and low-power storage of weights and inputs

Methodology Applied

Scientific EffectFerroelectric memory:

Data Source

PatentUS12265494B1Multi-die mapping matrix multiplication

Publication Date: 2025.04.01 KEPLER COMPUTING INC

AI summary

Matrix multiplication process is segregated between two separate dies—a memory die and a compute die to achieve low latency and high bandwidth artificial intelligence (AI) processor. The blocked matrix-multiplication scheme maps computations across multiple processor elements (PE) or matrix-multiplication units. The AI architecture for inference and training includes one or more PEs, where each PE includes memory (e.g., ferroelectric (FE) memory, FE-RAM, SRAM, DRAM, MRAM, etc.) to store weights and input/output I/O data. Each PE also includes a ring or mesh interconnect network to couple the PEs for fast access of information.