Neural Network Convolution Parallel Processing Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Dilated convolutional neural networks (CNNs) face memory bandwidth bottlenecks when processing large input feature maps, leading to performance issues due to the need for frequent memory accesses.
Innovation Solution
The proposed solution involves spatially dividing a large input feature map into smaller chunks and processing them in parallel across multiple computing engines (CEs). Each CE processes different portions of the input feature map, with overlapping intermediate states stored locally for sharing via an on-chip bus, eliminating the need for re-computation and reducing memory bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Area of stationary object
If dilated CNN processes large input feature maps, then it provides wider context and exponential receptive field increase, but it causes memory bandwidth bottlenecks due to frequent memory accesses
Solution Approach 1:
The input feature map is divided into multiple tiles, and each computing engine processes a specific tile independently. This segmentation allows parallel processing while reducing the memory bandwidth requirement for each individual engine, as each engine only needs to access its assigned tile and shared states from neighboring tiles rather than the entire large feature map.
Solution Approach 2:
The patent introduces a new dimension of parallelism by distributing processing across multiple computing engines (CEs) simultaneously. Instead of processing the large feature map sequentially in one engine, multiple CEs work in parallel on different tiles, effectively adding a spatial dimension to the computation that reduces the memory bandwidth burden on each individual processing unit.
2Productivity
If multiple computing engines process different tiles in parallel, then processing speed increases, but device complexity increases due to state sharing requirements
Solution Approach 1:
Adjacent computing engines share intermediate states through a state sharing mechanism that combines the processing results from overlapping tile regions. This merging approach allows CEs to reuse computations from neighboring tiles without re-computing shared states, reducing overall computational redundancy while maintaining parallel processing efficiency.
Solution Approach 2:
The patent introduces an intermediary state sharing mechanism that acts as a buffer between computing engines. This intermediary structure manages the exchange of intermediate states between adjacent CEs, allowing them to access shared computations without direct complex interconnections, thus simplifying the overall system architecture while enabling efficient parallel processing.
3Loss of time
If intermediate states are stored locally in each computing engine, then re-computation is eliminated, but on-chip memory usage increases
Solution Approach 1:
Each computing engine stores intermediate states locally in its own on-chip memory for quick access during processing. This local quality approach ensures that each CE has immediate access to its own intermediate states without needing to fetch them from external memory, eliminating re-computation delays while keeping the memory footprint localized to each engine's requirements rather than global.
Data Source
AI summary
Accelerated convolution of neural networks can be performed by executing N computing engines (CEs) of a neural network processor in parallel. An input dataset can be divided spatially into N chunks such that a respective last portion of each chunk overlaps with a respective first portion of a subsequent chunk. Portions of each chunk can be processed by a respective CE to generate a respective portion of an output dataset. The overlapping intermediate states computed by each CE from processing the overlapping portion can be stored locally for sharing with a subsequent CE using an on-chip bus.