Line Buffer Memory Shifting for Faster Neural Network Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural networks and deep neural networks face performance issues due to inefficient data processing and memory management, leading to latency and additional computations, as they do not effectively manage data operations across memory components, particularly in the line buffer.
Innovation Solution
The virtualization of hardware iterators in a neural network environment, utilizing a line buffer that shifts memory blocks by a calculated stride, allowing for contiguous data processing and optimized memory management, thereby improving performance by enabling efficient read/write operations and reducing computational cycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is processed from discontiguous memory locations in current neural networks, then memory management flexibility is improved, but processing speed and performance deteriorate due to additional computational cycles and latency
Solution Approach 1:
The patent introduces a line buffer as an intermediary component between discontiguous memory locations and the neural network processing units. The line buffer receives data from multiple scattered memory locations, reorganizes it into a contiguous format, and provides it to the processing units. This mediator resolves the contradiction by enabling flexible memory access patterns while maintaining continuous data flow for high-speed processing.
Solution Approach 2:
The system performs preliminary data reorganization in the line buffer before data reaches the neural network processing units. By pre-processing the data to convert it from discontiguous to contiguous format in advance, the system eliminates the need for additional computational cycles during the main processing phase, thus maintaining both flexibility and speed.
2Device complexity
If traditional memory access patterns are used in neural networks, then implementation simplicity is maintained, but productivity deteriorates due to avoidable processing time and memory operations
Solution Approach 1:
The patent segments the memory access function into two distinct parts: a simplified interface that neural network units see (contiguous memory access) and a complex backend implementation (discontiguous memory access with line buffer reorganization). This segmentation allows the neural network portion to remain simple while the line buffer handles the complex memory management, thereby improving productivity without significantly increasing overall system complexity.
3Adaptability or versatility
If neurons wait for data from discontiguous memory locations, then memory access flexibility is maintained, but time utilization deteriorates as neurons remain idle during processing cycles
Solution Approach 1:
The line buffer maintains continuous useful action by pre-fetching and reorganizing data from discontiguous memory locations before the neural network neurons need it. This ensures that neurons receive data continuously without idle waiting periods, while the system maintains flexible access to scattered memory locations through the buffer's data gathering capability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The performance of a neural network (NN) can be limited by the number of operations being performed. Using a line buffer that is directed to shift a memory block by a selected shift stride for cooperating neurons, data that is operatively residing memory and which would require multiple write cycles into a cooperating line buffer can be processed as in a single line buffer write cycle thereby enhancing the performance of a NN/DNN. A controller and/or iterator can generate one or more instructions having the memory block shifting values for communication to the line buffer. The shifting values can be calculated using various characteristics of the input data as well as the NN/DNN inclusive of the data dimensions. The line buffer can read data for processing, shift the data of the memory block and write the data in the line buffer for subsequent processing.