Neural Network Operator Execution Order for Constrained Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In constrained memory environments, such as mobile devices, the execution of neural networks can fail due to inadequate memory allocation, leading to potential data loss and device reboot, as the memory required for neural network execution often exceeds available memory limits.
Innovation Solution
A computer-implemented method and system that determines an optimal order of execution for neural network operators using graph search algorithms, identifying 'forced transitions' that do not increase peak memory consumption, and pruning the search space to ensure memory usage stays within available limits, thereby allowing static memory allocation and efficient data storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural network execution is performed with dynamic memory allocation, then flexibility in handling different computation scenarios is improved, but memory management complexity increases and may lead to exceeding available memory limits in constrained environments
Solution Approach 1:
The system performs preliminary analysis of the neural network computation graph to determine the optimal execution order and memory requirements before actual execution. This advance planning allows static memory allocation that prevents memory exhaustion while maintaining execution flexibility across different scenarios.
Solution Approach 2:
The system dynamically determines the execution order of operators in the neural network while using static memory allocation. This approach captures the essential dynamic aspect (execution ordering) while eliminating the complexity of dynamic memory allocation, achieving a balance between adaptability and simplicity.
2Manufacturing precision
If more memory is allocated for neural network execution, then computation accuracy and model complexity are improved, but the device may exceed its memory constraints and fail
Solution Approach 1:
The system performs preliminary analysis to calculate the exact memory requirements for each operator in the execution order. This advance computation allows allocation of precisely the right amount of memory needed for accurate computation without exceeding device constraints, thus maintaining both accuracy and reliability.
Solution Approach 2:
The system changes the execution order parameter of operators to optimize memory usage patterns. By reordering operations, the system can ensure that peak memory consumption stays within available limits while still allowing sufficiently complex models to execute with adequate accuracy.
3Quantity of substance
If the execution order of operators is optimized to reduce peak memory usage, then memory efficiency is improved, but the computation time may increase
Solution Approach 1:
The system performs preliminary optimization of the execution order to identify memory-efficient operator sequences. By pre-calculating the optimal order that minimizes peak memory usage, the system achieves memory efficiency without excessive computation time overhead, as the optimization is performed once before execution rather than during runtime.
Solution Approach 2:
The system adjusts the execution order parameter to balance memory efficiency and computation time. By carefully selecting operator sequences that reduce peak memory usage while maintaining reasonable computational flow, the system minimizes the trade-off between these two competing objectives.
4Quantity of substance
If graph search algorithms are used to determine optimal execution order, then memory usage optimization is improved, but the algorithm complexity and processing overhead increase
Solution Approach 1:
The system performs the graph search algorithm as a preliminary step before neural network execution to determine the optimal operator ordering. By completing this complex analysis in advance, the system achieves memory usage optimization while the processing overhead occurs during the setup phase rather than during actual inference, reducing runtime complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods are provided for determining an order of execution of a neural network. For instance, data indicative of a neural network and data indicative of an amount of available memory in a constrained memory space can be obtained. The neural network can include a plurality of operators. An order of execution associated with the neural network can then be determined. The order of execution specifies an order in which to execute each of the plurality of operators. The order of execution is determined based at least in part on the available memory in the constrained memory space. In particular, one or more graph search algorithms can be performed on a graph that is representative of the neural network to obtain the order of execution.