Shift Buffer Chaining for AI Accelerator Data Reuse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI hardware accelerators face inefficiencies in memory access and data duplication during convolution operations, affecting energy efficiency and performance, especially when dealing with repetitive memory access for the same input data.
Innovation Solution
The proposed accelerator includes a chaining buffer system with shift buffers and backup buffers that restructure input data for efficient processing within a systolic array structure, minimizing repetitive memory access and optimizing data reuse, thereby improving energy efficiency and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If input data is stored in memory and accessed repeatedly during convolution operations, then data availability is ensured, but memory access costs and energy consumption increase
Solution Approach 1:
The patent applies preliminary action by pre-loading input data into shift buffers before convolution operations begin. The shift buffers are pre-configured with input data in advance, allowing the systolic array to perform multiple convolution operations without repeated memory access. This is evident in the architecture where input data is loaded into shift buffers 220-1 through 220-N in advance of processing, enabling the PE array to operate continuously with cached data.
Solution Approach 2:
The patent introduces shift buffers as an intermediary component between memory and the systolic array. These buffers act as a mediator that holds input data temporarily, allowing the high-speed systolic array to access data without direct memory access. The shift buffers 220-1 through 220-N serve as intermediate storage that decouples the memory access rate from the processing rate, reducing energy consumption while maintaining data availability.
2Productivity
If shift buffers store and shift input data for processing, then data reuse is optimized, but device complexity increases due to additional buffer structures
Solution Approach 1:
The patent segments the buffering function into multiple specialized shift buffers (220-1 through 220-N), each dedicated to specific data streams or processing channels. This segmentation allows each buffer to be optimized for its specific function while collectively providing comprehensive data reuse capability. The segmented approach enables parallel operation of multiple buffers, improving productivity without requiring a single complex monolithic buffer structure.
Solution Approach 2:
The patent introduces a temporal dimension to data storage by implementing shift buffers that automatically advance through stored data over time. Instead of static storage, the buffers provide dynamic, time-dependent data access patterns that match the sequential processing requirements of convolution operations. This temporal organization enables efficient data reuse across multiple processing cycles without increasing spatial complexity.
3Duration of action of stationary object
If backup buffers store additional input data for row changes, then continuous processing is enabled, but loss of time occurs during data transfer and buffer switching
Solution Approach 1:
The patent applies preliminary action by pre-loading backup buffers with subsequent rows of input data before the current row processing completes. When a row change is needed, the system can immediately switch to pre-loaded data from backup buffers without waiting for memory access or performing time-consuming data transfers. This is implemented through backup buffers that are filled in advance with data needed for upcoming row transitions.
Solution Approach 2:
The patent ensures continuity of useful action by overlapping data preparation with data processing. While the systolic array processes current data, backup buffers simultaneously receive and prepare next-row data. This overlapping operation eliminates idle time during row transitions, as the useful action of data preparation continues uninterrupted alongside the useful action of data processing, maintaining continuous productive operation.
Data Source
AI summary
An accelerator includes: a memory configured to store input data; a plurality of shift buffers each configured to shift input data received sequentially from the memory in each cycle, and in response to input data being stored in each of internal elements of the shift buffer, output the stored input data to a processing element (PE) array; a plurality of backup buffers each configured to store input data received sequentially from the memory and transfer the stored input data to one of the shift buffers; and the PE array configured to perform an operation on input data received from one or more of the shift buffers and on a corresponding kernel.