Neural Network Accelerator Ping-Pong Buffer Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network-based systems, particularly convolution layers, are computationally intensive, making general-purpose CPUs inefficient for processing due to high latency and memory usage, which hinders performance in AI applications like computer vision and robotics.
Innovation Solution
A reconfigurable integrated circuit (IC) with a digital processing engine array and programmable logic, utilizing ping-pong buffers to concurrently process and store data, thereby hiding memory access latencies and optimizing data transfer, allowing for parallel processing of neural network layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a general-purpose CPU is used to execute neural network convolution layers, then the system has high flexibility and ease of operation, but the processing speed is slow and latency is high due to computational intensity
Solution Approach 1:
The system segments the neural network processing into distinct functional units: IO controllers for data input/output, feeding controllers for data preparation, multiple DPEs (Digital Processing Engines) for parallel computation, and ping-pong buffers for data staging. Each unit is optimized for its specific function, allowing the convolution layers to be processed in parallel across multiple DPEs while maintaining overall system coordination.
Solution Approach 2:
The patent introduces a new dimensional approach by adding multiple layers of buffering (IO buffers, feeding buffers, ping-pong buffers) between the CPU and computation units. This creates additional data flow dimensions that allow overlapping of memory access, data preparation, and computation operations, effectively hiding latency and improving throughput without increasing computational complexity.
2Productivity
If more data is stored in buffers to enable parallel processing, then processing throughput is improved, but memory usage and device complexity increase
Solution Approach 1:
The system performs preliminary actions by pre-loading data into ping-pong buffers before computation begins. The feeding controllers prepare and stage input data in feeding buffers while DPEs are configuring, and weight data is pre-loaded into weight buffers. This preliminary data preparation enables the computation units to immediately begin processing without waiting for memory access during computation, thereby improving throughput while managing memory usage through staged data loading.
3Loss of time
If ping-pong buffers are used to hide memory access latencies, then processing speed is improved, but device complexity and buffer management overhead increase
Solution Approach 1:
The patent merges the buffer management functions with the controller units. The IO controllers and feeding controllers are integrated with the ping-pong buffer management, allowing these controllers to simultaneously perform data transfer, buffer switching, and computation coordination. This merging reduces the need for separate buffer management logic and simplifies the overall system architecture while still achieving latency hiding through ping-pong buffering.
Data Source
AI summary
Examples herein describe hardware architecture for processing and accelerating data passing through layers of a neural network. In one embodiment, a reconfigurable integrated circuit (IC) for use with a neural network includes a digital processing engine (DPE) array, each DPE having a plurality of neural network units (NNUs). Each DPE generates different output data based on the currently processing layer of the neural network, with the NNUs parallel processing different input data sets. The reconfigurable IC also includes a plurality of ping-pong buffers designed to alternate storing and processing data for the layers of the neural network.