Neural Network Tile Overlap Buffer for Halo Data Reuse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional approaches to deep learning inference in neural networks waste computational resources by re-computing halo data between tiles and consume additional bandwidth by re-fetching halo data, leading to inefficiencies in memory access and computation.
Innovation Solution
Implementing an overlap data buffer to store portions of tiles between passes of chained layers, reducing the need to re-fetch and re-calculate halo data, using a secondary buffer to manage overlap data efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If halo data is re-computed for each tile, then computation is performed without additional storage, but computational resources are wasted
Solution Approach 1:
The patent segments the memory structure into a primary buffer and a secondary buffer (overlap buffer). The primary buffer stores current tile data while the secondary buffer stores overlap data from adjacent tiles. This segmentation allows efficient management of halo data without requiring complete recomputation, thereby reducing computational resource waste while maintaining a structured but manageable memory architecture.
Solution Approach 2:
The patent implements preliminary action by pre-fetching and storing overlap data (halo) from adjacent tiles into the secondary buffer before the main computation is needed. This allows the computational unit to directly access pre-computed overlap data during tile processing, eliminating the need to re-compute halo data and significantly reducing computational resource consumption.
2Loss of energy
If halo data is re-fetched from memory, then computation can proceed, but additional bandwidth is consumed
Solution Approach 1:
The patent merges the storage of overlap data into a dedicated secondary buffer that is integrated with the primary buffer system. By combining the overlap buffer with the main memory structure and managing both buffers through a unified control mechanism, the system reduces bandwidth consumption for halo data access while avoiding the need for completely separate memory systems, thus balancing bandwidth efficiency with manageable complexity.
Solution Approach 2:
The secondary buffer acts as an intermediary between the primary buffer and the computational unit. Instead of directly accessing and re-fetching halo data from the primary buffer or external memory for each tile computation, the system uses the secondary buffer as a mediator to store and provide overlap data, significantly reducing memory bandwidth consumption while simplifying the access pattern.
3Productivity
If overlap data is stored between passes, then computation utilization improves, but memory requirements increase
Solution Approach 1:
The patent applies local quality by creating a specialized secondary buffer with specific properties optimized for storing overlap data. This buffer has the exact capacity needed for halo data from adjacent tiles, providing localized storage with appropriate access patterns. This approach improves computation utilization by making overlap data immediately available while using only the necessary memory volume rather than general-purpose memory expansion.
Data Source
AI summary
Techniques for providing an overlap data buffer to store portions of tiles between passes of chained layers of a neural network are described. One accelerator circuit includes one or more processing units to execute instructions corresponding to the chained layers in multiple passes. In a first pass, the processing unit(s) receives a first input tile of an input feature map from a primary buffer and performs a first operation on the first input tile to obtain a first output tile. The processing unit stores the first output tile in the primary buffer and identifies a portion of the first output tile as corresponding to overlap data between tiles of the input feature map. The processing unit stores the portion in a secondary buffer. In a second pass, the processing unit retrieves the portion to avoid fetching the portion that overlaps and computing the overlap data again.