Accelerator Batch Sizing for On-Chip Memory-Limited Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network-based inference services face challenges in achieving low latency and efficiently processing multiple batches due to limited memory capacity and bandwidth, particularly with current CNN accelerators experiencing performance bottlenecks from external memory I/O.
Innovation Solution
A method involving dividing a neural network model into stages, determining maximum batch sizes processible in on-chip memory, and selecting a final batch size that minimizes computation and memory access costs to optimize batch processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple batches are processed using existing technologies, then throughput increases, but latency does not effectively decrease and memory bandwidth requirements exceed available resources
Solution Approach 1:
The patent segments the model execution into multiple stages and divides the processing into hierarchical batches (inner batches for on-chip memory and outer batches for off-chip memory). This segmentation allows the system to process multiple inner batches simultaneously within on-chip memory while managing outer batches sequentially, thereby increasing throughput without proportionally increasing latency.
Solution Approach 2:
The patent introduces a two-dimensional batch processing structure with inner batches (horizontal dimension) and outer batches (vertical dimension). Inner batches are processed in parallel within on-chip memory capacity, while outer batches are processed sequentially. This dimensional approach allows the system to maximize memory utilization and achieve high throughput while maintaining low latency for each individual batch.
2Productivity
If memory capacity and bandwidth are increased to process multiple batches, then processing capability improves, but resource constraints are violated
Solution Approach 1:
The patent implements a nested batch processing structure where inner batches are nested within outer batches. Multiple inner batches (e.g., 4 inner batches) are processed simultaneously within the limited on-chip memory capacity, and these are then nested within sequentially processed outer batches. This nesting allows the system to achieve high multi-batch processing capability while strictly adhering to the constrained memory capacity and bandwidth of on-chip resources.
3Productivity
If batch size is increased to improve throughput, then more data is processed per operation, but on-chip memory capacity is exceeded
Solution Approach 1:
The patent dynamically adjusts batch sizes based on the available on-chip memory capacity. The system calculates the maximum number of inner batches that can fit within on-chip memory for each model stage, and dynamically configures the batch processing parameters accordingly. This dynamic adaptation allows the system to maximize processing efficiency within the fixed on-chip memory capacity without requiring static over-provisioning of memory resources.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An operation method includes: dividing a model to be executed in an accelerator into a plurality of stages; determining, for each of the stages, a maximum batch size processible in an on-chip memory of the accelerator; determining the determined maximum batch sizes to each be a candidate batch size to be applied to the model; and determining, to be a final batch size to be applied to the model, one of the determined candidate batch sizes that minimizes a sum of a computation cost of executing the model in the accelerator and a memory access cost.