Tiled AI Accelerator With Activation Reuse for Lower Memory Bandwidth

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional AI accelerator circuits face memory bandwidth issues and delays due to excessive memory access, leading to inefficient computational performance and impractical power consumption and circuit complexity, especially with large AI models.

Innovation Solution

Implementing tiled MAC circuits with multiple processing elements that reuse input data through local storage and iterative propagation, reducing data duplication and memory access delays, and incorporating input registers and routing circuitry to enhance throughput and reduce power and area consumption.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional AI accelerator circuits perform computations with large AI models, then computational capability is improved, but memory bandwidth requirements increase excessively

Engineering Contradiction:

Improvecomputational capabilityVSAvoidmemory bandwidth

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent divides the AI accelerator into multiple processing elements (PEs) arranged in a grid, where each PE handles a specific portion of the computational task. This segmentation allows parallel processing of different data elements, reducing the total memory bandwidth required by distributing memory access responsibilities across multiple PEs rather than requiring a single PE to handle all data independently.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces shared memory as an intermediary storage mechanism between processing elements and external memory. This shared memory buffer allows PEs to reuse data without repeatedly accessing external memory, acting as a mediator that reduces memory bandwidth demands by enabling data to be stored and accessed locally across multiple computation steps.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If conventional AI accelerator circuits access memory frequently, then data availability is improved, but computation delays increase

Engineering Contradiction:

Improvedata availabilityVSAvoidcomputation delays

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent implements preliminary data loading into shared memory before computations begin. By pre-loading required data into the shared memory buffer, the system ensures data is immediately available to processing elements without causing computation delays during the actual computational phases.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent enables continuous computation by maintaining data in shared memory that can be reused across multiple computation steps. This eliminates idle waiting time between memory accesses and computations, as data remains available in the shared memory buffer for subsequent operations without requiring repeated external memory accesses.

Inventive Principle:

Principle #20Continuity of useful action

3Adaptability or versatility

If conventional AI accelerator circuits store duplicated data, then data reuse is improved, but power consumption increases

Engineering Contradiction:

Improvedata reuseVSAvoidpower consumption

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The patent implements a shared memory buffer that serves multiple processing elements simultaneously, enabling a single data storage location to function for multiple PEs. This universal storage approach allows data to be reused across different PEs without requiring each PE to maintain its own duplicate copies, thereby reducing overall power consumption while maintaining data reuse capabilities.

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Productivity

If conventional AI accelerator circuits increase processing elements, then computational throughput is improved, but circuit complexity increases

Engineering Contradiction:

Improvecomputational throughputVSAvoidcircuit complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent merges multiple processing elements into a unified grid architecture that shares common resources, particularly the shared memory buffer. By combining PEs that operate in parallel while accessing the same shared memory, the system achieves improved throughput without proportionally increasing circuit complexity, as the shared memory serves all PEs collectively rather than requiring separate dedicated storage for each PE.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS20250370714A1Tiled artificial intelligence accelerator with fine-grained activation reuse for minimized memory storage and access

Publication Date: 2025.12.04 TAIWAN SEMICONDUCTOR MANUFACTURING CO LTD

AI summary

Systems, devices, circuits, and methods of operating said systems, devices, and circuits are disclosed. In one aspect, a system includes an input buffer circuit storing a set of data values for a convolution operation and a plurality of multiply-accumulate (MAC) circuits. A first MAC circuit of the plurality of MAC circuits can retrieve the set of data values for the convolution operation and generate a first output by applying a first weight value stored at the first MAC circuit to a first data value of the set of data values. The first MAC circuit can provide the first data value to a second MAC circuit of the plurality of MAC circuits. The first MAC circuit can generate a plurality of second outputs by applying a second weight value and a third weight value stored at the first MAC circuit to a second data value of the set of data values.