Neural Network Layer Data Processing via Feature Map Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks, particularly convolutional neural networks, face significant computational costs and power consumption during inference and training due to repetitive convolutional computations, leading to underutilization of available compute resources in hardware accelerators like NPUs.
Innovation Solution
The method involves generating part-input feature maps (IFMs) that correspond to multiple original IFMs and convolving these part-IFMs with filters to produce part-output feature maps, allowing for increased utilization of compute resources by processing multiple part-IFMs in parallel, thereby improving operating performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If convolutional computations are performed repeatedly during inference or training, then computational accuracy is maintained, but processing time and power consumption increase significantly
Solution Approach 1:
The patent divides the input feature map into multiple tiles or partitions, allowing convolutional computations to be performed on smaller segments simultaneously. This segmentation enables parallel processing across multiple compute resources while maintaining the same computational accuracy as processing the complete feature map sequentially, thereby reducing overall processing time.
Solution Approach 2:
The patent introduces a spatial dimension for parallelization by processing different spatial partitions (tiles) of the feature map concurrently. Instead of performing computations sequentially in time, the solution distributes computations across multiple spatial partitions that can be processed in parallel by different compute resources, effectively trading time for spatial parallelism.
2Reliability
If convolutional computations are performed repeatedly during inference or training, then computational completeness is ensured, but power consumption increases due to bit switching in registers
Solution Approach 1:
By segmenting the feature map into tiles, the patent enables each compute resource to process only its assigned partition, reducing the amount of data that needs to be loaded and processed by each unit. This reduces the number of bit switches in registers while ensuring all computations are completed across the distributed system, thereby lowering overall power consumption.
Solution Approach 2:
The patent merges the computational tasks across multiple compute resources, where each resource handles a portion of the overall computation. This distributed approach reduces the computational burden on individual units, minimizing register activity and bit switching per resource, while the collective system maintains complete computational coverage.
3Speed
If specialized hardware accelerators like NPUs are used, then convolutional computation speed is improved, but compute resources remain underutilized
Solution Approach 1:
The patent segments the feature map into multiple tiles that can be independently processed by different compute resources within the NPU. This segmentation transforms a single computational task into multiple parallel tasks, enabling better utilization of the available compute resources while maintaining the high-speed convolutional computation capabilities of the specialized hardware.
Solution Approach 2:
By dividing the feature map into tiles, the patent ensures that compute resources can continuously process data without idle periods. Each compute resource has a dedicated tile to process, eliminating waiting time and ensuring continuous useful action across all hardware units, thereby maximizing resource utilization while maintaining high computation speed.
Data Source
AI summary
A method of processing input data for a given layer of a neural network using a data processing system comprising compute resources for performing convolutional computations is described. The input data comprises a given set of input feature maps, IFMs, and a given set of filters. The method comprises generating a set of part-IFMs including pluralities of part-IFMs which correspond to respective IFMs, of the given set of IFMs. The method further includes grouping part-IFMs in the set of part-IFMs into a set of selections of part-IFMs. The method further includes convolving, by respective compute resources of the data processing system, the set of selections with the given set of filters to compute a set of part-output feature maps. A data processing system for processing input data for a given layer of a neural network is also described.