Neural Network Processor Parallel Inference via Chunked Feature Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Dilated convolutional neural networks (CNNs) face performance bottlenecks due to large input feature maps that exceed on-chip memory, leading to memory bandwidth limitations and redundant computations when processing sequential data, such as speech synthesis and object detection, where the need for wider context requires larger receptive fields.
Innovation Solution
The proposed solution involves spatially dividing large input feature maps into smaller chunks and processing them in parallel across multiple computing engines (CEs) within a neural network processor, where overlapping intermediate states are stored locally and shared between CEs using an on-chip bus, eliminating redundant computations and memory bandwidth bottlenecks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If dilated CNNs are used to increase receptive fields for wider context, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent segments the input feature map into multiple smaller chunks that can be processed in parallel by multiple computing engines. Each chunk is processed independently through the dilated CNN layers, reducing the memory requirements for any single processing unit while maintaining the overall large receptive field coverage through parallel computation.
Solution Approach 2:
The patent introduces a new dimension of parallel processing by distributing chunks across multiple computing engines (CEs). Instead of processing the entire large feature map through a single sequential path, the system uses spatial parallelism across multiple engines to achieve both large receptive fields and reduced per-engine complexity.
2Manufacturing precision
If large input feature maps are processed to maintain resolution and coverage, then manufacturing precision is improved, but productivity deteriorates due to memory bandwidth limitations
Solution Approach 1:
The input feature map is divided into multiple smaller chunks that fit within on-chip memory capacities. This segmentation allows each computing engine to process chunks independently using local memory, eliminating the memory bandwidth bottleneck that would occur if the entire large feature map had to be transferred sequentially from off-chip memory.
Solution Approach 2:
The patent pre-divides the large input feature map into manageable chunks before processing begins. This preliminary segmentation allows the system to load only the necessary chunk data into on-chip memory at the start of processing, rather than requiring continuous large-bandwidth memory access during inference, thus improving processing speed while maintaining full feature map resolution.
3Device complexity
If sequential processing is used to handle large feature maps, then device complexity is reduced, but productivity deteriorates due to redundant computations
Solution Approach 1:
The patent segments both the input feature map and the processing workload across multiple computing engines. Each CE processes a specific chunk in parallel, eliminating the sequential bottleneck. The segmentation is designed so that overlapping regions are handled efficiently through shared state mechanisms, reducing redundant computations while maintaining architectural simplicity.
Solution Approach 2:
The patent merges the processing capabilities of multiple computing engines to handle the large feature map simultaneously. By combining the outputs from parallel chunk processing and integrating the shared state information, the system achieves high inference throughput without requiring a single complex sequential processor.
4Productivity
If parallel processing across multiple CEs is implemented to accelerate inference, then productivity is improved, but device complexity increases due to state management requirements
Solution Approach 1:
The patent introduces shared state buffers as intermediary structures that mediate between the parallel computing engines. These buffers store intermediate feature map representations that need to be accessed by multiple CEs, providing a standardized interface for state sharing without requiring complex direct communication between engines. This intermediary layer simplifies the coordination overhead while enabling efficient parallel processing.
Data Source
AI summary
A single neural network model can be used by each computing engine (CE) in a neural network processor to perform convolution operations in parallel for one or more stacks of convolutional layers. An input feature map can be divided into N chunks to be processed by N CEs, respectively. Each CE can process a last portion of a respective chunk to generate respective shared states to be used by a subsequent CE. A first CE uses pre-computed states to generate a first portion of an output feature map, while other CEs use shared states computed by a preceding CE to generate respective portions of the output feature map.