Near Memory Computing Accelerator for AI Data Bus Bottlenecks

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional computing systems face performance degradation, high power consumption, and data transfer bottlenecks when processing large AI models due to the need to frequently transmit weight data and intermediate data between non-volatile memory and computing circuits.

Innovation Solution

A hardware accelerator for machine learning computing systems is introduced, featuring a Near Memory Computing Unit (NMCU) that includes an input circuit, input decoder, weight decoder, product engine circuit, quantization logic, and control logic. This setup allows for efficient data processing by fetching weights directly from non-volatile memory and using a ping-pong buffer to minimize data bus usage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data is stored in non-volatile flash memory, then storage capacity per cost is improved, but data transfer bottleneck occurs on data bus

Engineering Contradiction:

Improvestorage capacityVSAvoiddata transfer speed

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent combines storage functionality and computing functionality into a single integrated architecture. Non-volatile memory cells are directly coupled to compute units, allowing weight data to be stored and processed in the same location, eliminating the need for separate data transfer between storage and computing components.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent introduces an input circuit with buffer circuits as an intermediary between the host device and the non-volatile memory. This buffer circuit temporarily stores input data and weight data, facilitating efficient data exchange without creating bottlenecks on the main data bus.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If weight data is transmitted multiple times from non-volatile memory to computing circuit, then computation accuracy is maintained, but power consumption increases

Engineering Contradiction:

Improvecomputation accuracyVSAvoidpower consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent performs preliminary action by pre-loading weight data into buffer circuits within the input circuit before computation begins. This allows the computing unit to access weight data locally without repeated transmissions from non-volatile memory, reducing power consumption while maintaining computation accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent merges storage and computing functions by integrating buffer circuits directly with the computing unit. This combination allows weight data to be retained in proximity to the compute unit throughout the computation process, eliminating repeated data transfers and associated power consumption.

Inventive Principle:

Principle #5Merging (Combining)

3Ease of operation

If data is stored in DRAM for refresh operation, then data accessibility is improved, but power consumption increases

Engineering Contradiction:

Improvedata accessibilityVSAvoidpower consumption

Core Design Contradiction:

Ease of operationVSUse of energy by moving object

Solution Approach 1:

The patent applies local quality by using different memory technologies for different purposes: non-volatile flash memory for long-term weight data storage (where persistence is critical) and DRAM buffer circuits for temporary input data buffering (where fast access is critical). This localized approach optimizes both accessibility and power consumption for each specific function.

Inventive Principle:

Principle #3Local quality

4Quantity of substance

If vast amounts of parameter data are stored for large AI models, then model capability is improved, but data bus bottleneck worsens

Engineering Contradiction:

Improveparameter capacityVSAvoiddata processing throughput

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent segments the data handling architecture into distinct functional blocks: non-volatile memory for weight data, input circuit with buffer circuits for input data management, and computing units for processing. This segmentation allows parallel operation of multiple data streams and eliminates the single data bus bottleneck by providing dedicated data paths for different types of data.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20250130805A1Non-volatile memory based near-memory computing machine learning accelerator

Publication Date: 2025.04.24 SEMIBRAIN INC

AI summary

A computing device comprising: an input circuit to receive and buffer intermediate inputs being processed by the computing device and inputs transferred between a host device and the computing device; an input decoder to fetch inputs from the input circuit and arrange corresponding input data with n elements into a specific memory address; a weight decoder, coupled to the non-volatile memory device with a plurality of high bandwidth data bus, to fetch weights from the non-volatile memory and arrange corresponding weight data with n elements into a specific memory address; a product engine circuit with a group of dot product engines for taking the input data and the weight data, and returning weighted sum; a 10 quantization logic to quantize the weighted sum into a set of discrete values; and a control logic circuit to selectively enable or disable data elements of each data input data a specific number of times.