Pre-pooler Circuit for Neural Network Tensor Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Nested neural networks face inefficiencies due to data dependencies between branches, leading to idle hardware resources as some elements wait for upstream operations to complete, particularly in convolution and pooling operations.
Innovation Solution
Incorporating a dedicated pre-pooler and a 2R3W tensor buffer to perform pre-pooling operations in parallel with other neural network layers, reducing the idle time of the multiply-add array by allowing simultaneous access and processing of tensor data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data dependencies are enforced between layers in nested neural networks, then computational accuracy is maintained, but hardware resource utilization deteriorates due to idle periods
Solution Approach 1:
The patent implements pre-pooling operations that process tensor data in advance before it reaches the multiply-add array. By performing pooling operations preliminarily on tensor data stored in the buffer, the system prepares processed data ready for immediate use by the multiply-add array, eliminating idle waiting periods while maintaining the required computational sequence and accuracy
Solution Approach 2:
The patent introduces a temporal dimension to the data processing by implementing multi-page buffer architecture with parallel read/write operations. Different pages of the tensor buffer can be accessed simultaneously for different operations, allowing the system to bypass sequential data dependency constraints while maintaining computational integrity through structured data flow management
2Reliability
If sequential processing is used to maintain data dependencies, then computational correctness is ensured, but execution time increases due to idle hardware elements
Solution Approach 1:
The pre-pooling mechanism performs pooling operations in advance on tensor data before it is needed by subsequent layers. This preliminary processing eliminates waiting time for the multiply-add array while ensuring that data is processed in the correct computational sequence, thus reducing execution time without sacrificing correctness
Solution Approach 2:
The patent enables continuous operation of hardware elements by implementing overlapping execution of different operations on different data pages. The multiply-add array can continuously process data from one page while pooling operations simultaneously prepare data for another page, eliminating idle periods and maintaining continuous useful action throughout the system
3Productivity
If dedicated pre-pooling hardware is added, then parallel processing capability is improved, but device complexity increases
Solution Approach 1:
The pre-pooling hardware is designed with multi-functionality to perform different pooling operations (max pooling, average pooling, etc.) and can process multiple tensor pages simultaneously. This universal design reduces the need for separate dedicated hardware for each operation type, thereby limiting the increase in device complexity while maintaining parallel processing capabilities
Solution Approach 2:
The pre-pooling operations are nested within the existing neural network processing architecture, with the pre-pooler integrated into the data flow between tensor buffer and multiply-add array. This nested structure allows the pre-pooling functionality to be incorporated without requiring a complete architectural redesign, thus limiting complexity increase while enabling parallel processing
Data Source
AI summary
Embodiments herein describe circuitry with improved efficiency when executing layers in a nested neural network. As mentioned above, a nested neural network has at least one split operation where a tensor generated by a first layer is transmitted to, and processed by several branches in the neural network. Each of these branches can have several layers that have data dependencies which result in a multiply-add array sitting idly. In one embodiment, the circuitry can include a dedicated pre-pooler for performing a pre-pooling operation. Thus, the pre-pooling operation can be performing in parallel with other operations (e.g., the convolution performed by another layer). Once the multiply-add array is idle, the pre-pooling operation has already completed (or at least, has already started) which means the time the multiply-add array must wait before it can perform the next operation is reduced or eliminated.