Line Buffer Shift Strides for Contiguous Neural Network Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current artificial neural networks (NNs) and deep neural networks (DNNs) face performance issues due to inefficient data processing and memory management, leading to latency and unnecessary computations, particularly in managing data across neural processing components.
Innovation Solution
The implementation of virtualized hardware iterators and a cooperating line buffer component that allows for optimized memory management by shifting memory blocks in a single processing cycle, enabling the extraction of data from multiple overlapping sliding windows as contiguous memory blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is processed from discontiguous memory blocks using conventional methods, then memory management is straightforward, but processing time increases and performance decreases
Solution Approach 1:
The patent introduces a line buffer as an intermediary component between discontiguous memory blocks and the neural processing units. The line buffer receives data from multiple discontiguous memory locations, reorganizes it into contiguous format through shift operations, and provides it to processing units. This mediator eliminates the need for processors to handle discontiguous data directly, thereby reducing processing time and improving throughput without changing the underlying memory architecture.
Solution Approach 2:
The system performs preliminary actions by pre-fetching data from discontiguous memory blocks into the line buffer before actual neural network processing begins. The line buffer proactively reorganizes this data into contiguous format and makes it ready for processing. This preliminary reorganization eliminates wait times during computation, as processing units receive continuously available data without needing to pause for memory access operations.
2Productivity
If conventional memory access methods are used, then hardware complexity remains low, but processing efficiency deteriorates due to multiple memory reads
Solution Approach 1:
The patent merges multiple discontiguous memory reads into a single unified operation by using the line buffer to collect data from various memory locations and present it as a single contiguous block to processing units. This combining approach reduces the total number of read operations required, decreases memory management complexity, and improves processing throughput by eliminating redundant access overhead.
3Productivity
If data is not pre-organized in contiguous blocks, then memory usage is optimized for storage, but computational performance suffers due to inefficient data extraction
Solution Approach 1:
The line buffer serves as an energy-efficient intermediary that performs shift operations in a dedicated buffer region rather than requiring repeated high-energy memory access operations. By keeping the intermediary buffer on-chip or in fast memory close to processing units, the system minimizes energy-consuming data transfers while maintaining optimized storage layouts in main memory.
Solution Approach 2:
The system performs preliminary data reorganization into contiguous format in the line buffer before computational processing begins. This preliminary action ensures that when neural network operations execute, data is already in the optimal contiguous format required for efficient processing, eliminating the need for energy-intensive data movement operations during computation and reducing overall power consumption.
Data Source
AI summary
The performance of a neural network (NN) can be limited by the number of operations being performed. Using a line buffer that is directed to shift a memory block by a selected shift stride for cooperating neurons, data that is operatively residing memory and which would require multiple write cycles into a cooperating line buffer can be processed as in a single line buffer write cycle thereby enhancing the performance of a NN/DNN. A controller and/or iterator can generate one or more instructions having the memory block shifting values for communication to the line buffer. The shifting values can be calculated using various characteristics of the input data as well as the NN/DNN inclusive of the data dimensions. The line buffer can read data for processing, shift the data of the memory block and write the data in the line buffer for subsequent processing.