Tiled ASIC Systolic Array for Low-Latency Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for neural networks perform computations for convolutional layers in software, leading to inefficiencies in data transfer and increased latency due to the need for software-based operations such as dot product calculations and activation function applications, which can hinder the performance of machine learning models.
Innovation Solution
An application-specific integrated circuit (ASIC) chip with a systolic array of cells arranged in tiles, multiple controllable bus lines, and a vector processing unit, which enables efficient data transfer and computation by performing operations like multiplication and addition in hardware, reducing latency through optimized data paths and local memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If computations for convolutional layers are performed in software, then flexibility and ease of implementation are maintained, but computation speed and latency are degraded
Solution Approach 1:
The patent replaces software-based computation mechanisms with hardware-based systolic array architecture. The systolic array uses dedicated computational cells that perform multiply-accumulate operations in parallel, substituting the sequential software execution model with a parallel hardware model, thereby achieving significant speedup for convolutional neural network operations
Solution Approach 2:
The computational array is segmented into multiple tiles, where each tile contains a subarray of computational cells and local memory. This segmentation allows for distributed computation and memory access, enabling parallel processing while managing hardware complexity through modular design
2Quantity of substance
If data is transferred between computational units using standard bus structures, then ease of implementation is maintained, but bandwidth and latency are degraded
Solution Approach 1:
The patent transitions from traditional one-dimensional bus structures to a two-dimensional data path network within the tile architecture. Data can flow horizontally and vertically between computational cells and memory units, providing multiple concurrent data transfer paths and significantly increasing effective bandwidth without requiring a single complex high-speed bus
3Loss of time
If memory is centralized rather than distributed, then device complexity is reduced, but access time and latency increase
Solution Approach 1:
Memory is segmented and distributed to each computational tile, with each tile containing its own local memory units. This allows computational cells within a tile to access their required data locally without waiting for centralized memory access, dramatically reducing memory access latency while maintaining manageable complexity through modular replication
4Productivity
If software-based operations are used for dot product calculations and activation functions, then adaptability is maintained, but processing speed is degraded
Solution Approach 1:
The patent substitutes software-based mathematical operations with dedicated hardware circuits within the systolic array cells. Each computational cell contains hardware multipliers and adders for dot product calculations, and the architecture supports configurable activation functions implemented in hardware, achieving high-speed processing while maintaining operational flexibility through programmable control
Data Source
AI summary
An application specific integrated circuit (ASIC) chip includes: a systolic array of cells; and multiple controllable bus lines configured to convey data among the systolic array of cells, in which the systolic array of cells is arranged in multiple tiles, each tile of the multiple tiles including 1) a corresponding sub array of cells of the systolic array of cells, 2) a corresponding subset of controllable bus lines of the multiple controllable bus lines, and 3) memory coupled to the subarray of cells.