Pipelined Heterogeneous Dataflow for AI Accelerator Throughput

Solution Overview

Problem

Existing AI accelerators are limited by a single fixed dataflow that does not optimize performance for diverse neural network layer types, such as convolutional and fully connected layers, leading to suboptimal efficiency and throughput.

Innovation Solution

Implementing a pipelined architecture with separate convolutional and fully connected cores, each optimized for weight stationary and input stationary dataflows, respectively, to improve latency and throughput by reducing interconnect overhead and enhancing PE utilization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If a single fixed dataflow architecture is used, then device complexity is reduced, but productivity and adaptability to different neural network layer types deteriorate

Engineering Contradiction:

Improvedataflow architecture complexityVSAvoidthroughput

Core Design Contradiction:

Device complexityVSProductivity

Solution Approach 1:

The accelerator is divided into separate processing cores: a first processing core for convolutional layers with weight stationary dataflow, and a second processing core for fully connected layers with input stationary dataflow. Each core is optimized for its specific layer type, eliminating the need for a single complex universal architecture while improving overall throughput and adaptability.

Inventive Principle:

Principle #1Segmentation

2Device complexity

If a single fixed dataflow architecture is used, then device complexity is reduced, but adaptability to diverse neural network layer types deteriorates

Engineering Contradiction:

Improvedataflow architecture complexityVSAvoidadaptability to layer types

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

The accelerator is divided into separate processing cores: a first processing core for convolutional layers with weight stationary dataflow, and a second processing core for fully connected layers with input stationary dataflow. Each core is optimized for its specific layer type, eliminating the need for a single complex universal architecture while improving overall throughput and adaptability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The accelerator achieves universality by implementing multiple dataflow architectures within a single device - both weight stationary and input stationary dataflows are supported through separate processing cores. This allows the accelerator to efficiently handle diverse neural network layer types (convolutional and fully connected) without requiring external hardware changes, thereby improving adaptability while maintaining manageable complexity through modular design.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Productivity

If separate pipelined cores are implemented, then productivity and PE utilization improve, but device complexity increases

Engineering Contradiction:

ImprovethroughputVSAvoidpipeline architecture complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The accelerator is divided into separate processing cores: a first processing core for convolutional layers with weight stationary dataflow, and a second processing core for fully connected layers with input stationary dataflow. Each core is optimized for its specific layer type, eliminating the need for a single complex universal architecture while improving overall throughput and adaptability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The pipelined architecture enables continuous processing where the first processing core processes convolutional layers while the second processing core simultaneously processes fully connected layers. This continuity of useful action across multiple cores improves overall throughput and productivity by eliminating idle time and improving resource utilization.

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS20250328762A1System and methods for piplined heterogeneous dataflow for artificial intelligence accelerators

Publication Date: 2025.10.23 TAIWAN SEMICONDUCTOR MANUFACTURING CO LTD

AI summary

Systems and methods for a pipelined heterogeneous dataflow for an artificial intelligence accelerator are disclosed. A pipelined processing core includes a first processing core configured to have a first type of dataflow and a second processing core configured to have a second type of dataflow. The first processing core includes a matrix array of PEs arranged in columns and rows, each of the PEs configured to perform a MAC operation based on an input and a weight. The second processing core is configured to receive an output from the first processing core. The second processing core includes a column of PEs configured to perform MAC operations.