Neural Network Layer Pipeline With Line Buffers for Lower Memory Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network computing devices face high storage overheads due to the need to store all computation results from previous layers before proceeding, leading to inefficient use of resources and increased computational time.
Innovation Solution
Implementing an inter-layer pipeline control mechanism that allows the next neural network layer to start computation only when the previous layer has stored sufficient data elements, reducing the need to store all intermediate data and optimizing storage usage by using a line buffer with N registers to buffer data between layers at minimum storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all intermediate data from previous layers is stored before proceeding to the next layer, then computation accuracy is maintained, but storage overhead increases and computation time increases
Solution Approach 1:
The patent segments the intermediate data storage into multiple line buffers (first line buffer, second line buffer, third line buffer) instead of using a single large storage unit. Each line buffer stores a portion of the intermediate data, allowing the system to process and clear data in segments. This segmentation reduces the peak storage requirement while maintaining the ability to access necessary intermediate values for computation accuracy.
Solution Approach 2:
The patent implements preliminary action by pre-allocating and organizing line buffers before computation begins, and by pre-clearing buffers after their data is consumed. The control unit manages the timing and sequence of buffer operations, ensuring that data is prepared in advance for the next computational stage while freeing up storage space proactively, thus reducing overall storage overhead.
2Reliability
If all intermediate data from previous layers is stored before proceeding to the next layer, then complete data availability is ensured, but computation time increases
Solution Approach 1:
The patent enables continuous computation by implementing an overlapping execution model where the second computing unit can begin processing with data from the first line buffer while the first computing unit continues to generate and store intermediate data in subsequent buffers. This continuity eliminates idle waiting time between layers, as the pipeline remains full of useful computational actions throughout the process.
Solution Approach 2:
The control unit performs preliminary actions by pre-managing buffer allocation and data transfer timing, ensuring that the second computing unit receives necessary data at the optimal moment without waiting for complete computation of all intermediate layers. This proactive data management reduces computation time while maintaining data availability.
3Quantity of substance
If a large quantity of storage resources is allocated for intermediate data, then storage capacity requirements are met, but device complexity and resource efficiency deteriorate
Solution Approach 1:
The patent divides the storage system into multiple manageable line buffers with fixed, predetermined sizes based on computational parameters (h, w, W, p). This segmentation transforms a complex, large-scale storage problem into several simple, standardized buffer units, reducing device complexity while providing sufficient storage capacity for intermediate data.
Solution Approach 2:
The patent determines buffer sizes using specific parameter relationships: N=(h-1)×(W+p)+w for the first line buffer and M=h×(W+p) for the second line buffer, where h, w, W, and p represent computational parameters. By changing storage allocation from fixed large blocks to dynamically sized buffers based on actual computational needs, the system reduces both storage overhead and device complexity.
4Reliability
If the next layer waits for complete computation of the previous layer, then data consistency is maintained, but clock cycle waste increases
Solution Approach 1:
The control unit performs preliminary actions by pre-managing data transfer timing between computing units and buffers. It ensures that the second computing unit receives consistent, validated data from the first computing unit at the precise moment when it becomes available, rather than waiting for complete layer computation. This proactive timing management maintains data consistency while eliminating idle clock cycles.
Solution Approach 2:
The patent establishes continuous data flow between computing units through the buffer system, where the second computing unit continuously receives and processes data as it becomes available from the first computing unit. This continuous operation eliminates gaps and idle clock cycles while maintaining data consistency through controlled buffer management and synchronized data transfer.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
A computing device (300) and a computation method for neural network computation are disclosed. The computing device (300) includes: a first computing unit (310), configured to perform a first operation on an input first matrix M times, to obtain a second matrix; a second computing unit (330), configured to perform a second operation on the input second matrix; and a control unit (340), configured to: control the first computing unit (310) to perform an ith first operation of the M first operations on the first matrix, to obtain an ith data element of the second matrix; store the ith data element of the second matrix into a first storage unit (320); and control, if data elements currently stored in the first storage unit (320) are sufficient for performing one second operation, the second computing unit (330) to perform one second operation. The computing device (300) and the computation method can reduce storage overheads of the computing device (300) for neural network computation.