Dynamic Neural Network Batching Controller for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing batching methods for neural network workloads rely on static batch sizes, which can lead to inefficiencies and increased latency due to mismatched computing and memory cycles across different layers of a neural network topology.
Innovation Solution
A dynamic batching controller determines an optimal batch size for each layer of a neural network topology by analyzing the number of operations and weights of each layer, ensuring that the batch size aligns with the computing resources available, thereby reducing processing latency and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If static batch size is used for neural network workloads, then implementation simplicity is maintained, but processing latency increases due to mismatched computing and memory cycles across different layers
Solution Approach 1:
The patent implements dynamic batching that adjusts batch sizes adaptively based on layer-specific characteristics. The system transitions from static batch sizing to dynamic batch sizing where the batch size varies across different layers of the neural network topology, optimizing the match between computing cycles and memory cycles for each layer.
Solution Approach 2:
The patent applies different batch sizes to different layers of the neural network based on their specific characteristics. Each layer receives a customized batch size that matches its computing and memory requirements, rather than applying a uniform batch size across all layers.
2Use of energy by stationary object
If static batch size is used for neural network workloads, then system complexity remains low, but power consumption increases due to inefficient resource utilization
Solution Approach 1:
The system dynamically adjusts batch sizes to match the actual computing and memory requirements of each layer, preventing both underutilization and overutilization of resources. This dynamic adaptation optimizes power consumption by ensuring that computing resources are efficiently utilized without waste.
Solution Approach 2:
The patent changes the batch size parameter based on layer-specific characteristics such as the number of operations and weights. By adjusting this key parameter dynamically, the system optimizes resource utilization and reduces power consumption.
3Productivity
If larger batch size is used to improve throughput, then processing speed increases, but memory bandwidth requirements increase causing bottlenecks
Solution Approach 1:
The patent applies different batch sizes to different layers based on their specific memory and computing requirements. Layers with higher memory bandwidth requirements receive smaller batch sizes, while layers with lower requirements can handle larger batch sizes, thereby optimizing overall throughput without creating memory bottlenecks.
Data Source
AI summary
Examples to determine a dynamic batch size of a layer are disclosed herein. An example apparatus to determine a dynamic batch size of a layer includes a layer operations controller to determine a layer ratio between a number of operations of a layer and weights of the layer, a comparator to compare the layer ratio to a number of operations per unit of memory size performed by a computation engine, and a batch size determination controller to, when the layer ratio is less than the number of operations per unit of memory size, determine the dynamic batch size of the layer.