Neural Network Memory Scheduling for Low-Buffer Edge Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Edge devices with limited computational and memory resources face challenges in efficiently executing neural networks due to large memory requirements for accumulating neural node outputs and managing multiple computational paths, leading to suboptimal utilization of memory buffers.
Innovation Solution
Optimize memory usage by evaluating different orders of branch execution in neural networks to minimize memory requirements, using alternating buffers for odd and even layers, interrupting branch execution temporarily, and rescaling output values in batches to reduce buffer size, along with dynamic integer formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If standard neural network execution is used on edge devices, then computational accuracy is maintained, but memory usage becomes excessive
Solution Approach 1:
The patent segments the accumulation of neural node outputs into multiple batches, processing them in sequential steps rather than accumulating all outputs simultaneously. This division allows memory usage to be reduced to the size of individual batches rather than the total size of all outputs, making neural network execution feasible on edge devices with limited memory resources while maintaining computational accuracy through complete processing of all batches
Solution Approach 2:
The patent performs preliminary determination of an optimal execution order for parallel branches before actual computation. By analyzing dependencies and memory requirements in advance, the system arranges the execution sequence to minimize peak memory usage, allowing less memory-intensive operations to be performed first and freeing memory for subsequent operations, thereby reducing overall memory requirements while preserving computational correctness
2Productivity
If parallel branches are executed simultaneously, then computational speed is improved, but memory demand increases
Solution Approach 1:
The patent dynamically adjusts the execution strategy for parallel branches based on memory availability and computational dependencies. Instead of fixed simultaneous or sequential execution, the system flexibly interleaves parallel branch execution with memory management, allowing computational speed to be maintained where possible while adapting to memory constraints by pausing and resuming branches as memory becomes available
Solution Approach 2:
The patent implements periodic memory checks and buffer management during the execution of parallel branches. By periodically freeing buffers after intermediate results are obtained and allocating memory in controlled intervals, the system maintains higher computational throughput compared to fully sequential execution while preventing memory exhaustion that would occur with completely simultaneous execution of all parallel branches
Data Source
AI summary
Implementations disclosed describe methods and systems to perform the methods of optimizing a size of memory used for accumulation of neural node outputs and for supporting multiple computational paths in neural networks. In one example, a size of memory used to perform neural layer computations is reduced by performing nodal computations in multiple batches, followed by rescaling and accumulation of nodal outputs. In another example, execution of parallel branches of neural node computations include evaluating, prior to the actual execution, the amount of memory resources needed to execute a particular order of branches sequentially and select the order that minimizes this amount or keeps this amount below a target threshold.