Neural Network Memory Scheduling for Batched Branch Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Edge devices with limited computational and memory resources face challenges in efficiently executing neural networks due to large memory requirements for accumulating neural node outputs and handling multiple computational paths, leading to suboptimal utilization of memory buffers.
Innovation Solution
The system optimizes memory usage by evaluating different orders of branch execution in neural networks to minimize memory requirements, using alternating buffers for odd and even layers, rescaling output values in batches, and dynamically adjusting the number of batches based on specific layer outputs, while reusing memory portions and applying run-time optimization techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If memory buffers are allocated to accumulate all neural node outputs simultaneously, then complete computational paths are supported, but memory footprint increases significantly
Solution Approach 1:
The patent segments the accumulation of neural node outputs into multiple batches instead of accumulating all outputs simultaneously. Each batch processes a subset of outputs, allowing memory buffers to be reused across batches. This segmentation reduces peak memory requirements while maintaining support for complete computational paths through sequential processing.
Solution Approach 2:
The patent implements periodic action by processing neural node outputs in repeated batches across multiple passes. Memory buffers are allocated for one batch, processed, then reused for the next batch. This periodic batched processing pattern enables complete computational path support while keeping memory footprint bounded by buffer size rather than total output volume.
2Measurement precision
If multiple computational paths are handled simultaneously with dedicated buffers, then execution accuracy is maintained, but memory utilization becomes suboptimal
Solution Approach 1:
The patent merges the handling of multiple computational paths by using shared memory buffers that serve multiple branches sequentially within each batch. Instead of allocating dedicated buffers for each computational path, the same buffers are reused across different paths and batches, improving memory utilization while maintaining execution accuracy through proper data management and ordering.
Solution Approach 2:
The patent introduces dynamic batch processing where the number and size of batches can be adjusted based on memory constraints and computational requirements. This dynamic approach allows the system to adapt memory allocation to actual needs, optimizing the balance between accuracy and utilization efficiency rather than using static dedicated buffers for each path.
3Quantity of substance
If batch processing is used to reduce memory requirements, then memory footprint decreases, but computational overhead increases
Solution Approach 1:
The patent changes the parameter of batch size to optimize the trade-off between memory footprint and computational overhead. By adjusting batch size parameters, the system can reduce memory requirements through smaller batches while minimizing the overhead impact. The batch size is tuned to achieve acceptable memory reduction without excessive fragmentation or management complexity.
Data Source
AI summary
Implementations disclosed describe methods and systems to perform the methods of optimizing a size of memory used for accumulation of neural node outputs and for supporting multiple computational paths in neural networks. In one example, a size of memory used to perform neural layer computations is reduced by performing nodal computations in multiple batches, followed by rescaling and accumulation of nodal outputs. In another example, execution of parallel branches of neural node computations include evaluating, prior to the actual execution, the amount of memory resources needed to execute a particular order of branches sequentially and select the order that minimizes this amount or keeps this amount below a target threshold.