Neural Network Processor Batch Processing Circuit
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional neural network systems compute inferences serially, which is inefficient when processing multiple inputs, as each input must be processed through all layers before the next input can be processed.
Innovation Solution
A special-purpose hardware circuit that processes multiple neural network inputs in parallel by forming batches of inputs based on a weight reuse value, allowing weight inputs to be reused across multiple independent inputs, thereby maximizing throughput and minimizing memory access wait times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network systems compute inferences serially, then each input is processed through all layers to generate the output, but the processing speed and throughput are reduced
Solution Approach 1:
The patent segments the neural network processing into independent layers that can be processed in parallel. Each layer is treated as an independent computational unit that can operate simultaneously on multiple inputs, breaking the sequential dependency between layers while maintaining the computational integrity of each layer
Solution Approach 2:
The patent introduces batch processing as an additional dimension of parallelism. Instead of processing inputs sequentially through layers, the system processes batches of inputs through layers simultaneously, adding a temporal dimension to the processing pipeline that enables parallel execution of multiple inference operations
2Productivity
If weight inputs are reused across multiple inputs, then throughput is maximized, but memory access patterns become more complex
Solution Approach 1:
The patent applies preliminary action by pre-loading weight inputs into on-chip memory before processing batches of inputs. This allows the computation units to access weights from fast on-chip memory rather than external memory during processing, reducing memory access complexity and improving throughput
Solution Approach 2:
The patent ensures continuity of useful action by maintaining weight inputs in on-chip memory throughout the batch processing operation. This allows continuous computation without repeated external memory accesses, maximizing the utilization of computation units and minimizing idle time
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a respective neural network output for each of a plurality of inputs, the method comprising, for each of the neural network layers: receiving a plurality of inputs to be processed at the neural network layer; forming one or more batches of inputs from the plurality of inputs, each batch having a number of inputs up to the respective batch size for the neural network layer; selecting a number of the one or more batches of inputs to process, where a count of the inputs in the number of the one or more batches is greater than or equal to the respective associated batch size of a subsequent layer in the sequence; and processing the number of the one or more batches of inputs to generate the respective neural network layer output.