Tensor Rematerialization and Paging for Neural Network Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The deployment and training of large neural network models on edge devices are hindered by limited memory capacity and high latency, as they require significant computational resources and memory, leading to challenges in efficient deployment and inference on resource-limited devices.
Innovation Solution
An apparatus and method that determine retention intervals for node outputs in local memory based on rematerialization and paging constraints, optimizing the sequence of tasks to reduce peak memory utilization and inference latency by jointly optimizing tensor rematerialization and paging.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural networks are deployed on edge devices, then accessibility and application scope are improved, but memory capacity limitations and high latency worsen performance
Solution Approach 1:
The patent segments the computational graph into multiple stages and divides tensor retention management into discrete retention intervals. Each tensor is assigned specific retention periods in local memory, allowing the system to process large neural networks by breaking down the memory management into manageable segments rather than requiring all tensors to reside in memory simultaneously.
Solution Approach 2:
The patent implements a retention interval mechanism where tensor outputs are automatically discarded from local memory after their designated retention period expires. This allows the system to recover local memory capacity for subsequent tensors, enabling efficient processing of large neural networks with limited memory by continuously discarding and recovering memory space.
2Reliability
If complex neural networks are used to improve model accuracy, then performance is improved, but memory footprint and computational requirements worsen resource utilization
Solution Approach 1:
The patent introduces dynamic retention interval assignment where the retention period of each tensor is adaptively determined based on its usage patterns and dependencies in the computational graph. This dynamic approach allows complex neural networks to be executed efficiently by optimizing memory retention periods in real-time, reducing overall memory footprint while maintaining model accuracy.
Solution Approach 2:
The patent changes the parameter of tensor retention time from a static value to a dynamically determined interval based on computational dependencies. By adjusting retention parameters adaptively, the system can accommodate complex network architectures with varying memory requirements at different computation stages, reducing peak memory usage while preserving model performance.
3Speed
If all node outputs are retained in local memory to ensure fast access, then access speed is improved, but memory capacity is exceeded
Solution Approach 1:
The patent performs preliminary analysis of the computational graph to determine retention intervals for each tensor before execution. By pre-calculating which tensors need to be retained and for how long, the system optimizes local memory usage from the outset, ensuring fast access to necessary tensors while avoiding memory overflow by not retaining unnecessary data.
Solution Approach 2:
The patent extracts and identifies critical tensors that must be retained in local memory based on their dependency relationships in the computational graph. By separating essential tensors from non-essential ones, the system retains only the necessary data in fast local memory, maintaining access speed for critical operations while reducing overall memory capacity requirements.
4Productivity
If retention intervals are extended to reduce rematerialization, then computational efficiency is improved, but memory usage increases
Solution Approach 1:
The patent applies partial retention by assigning retention intervals only to the specific tensors that are actually needed for subsequent computations, rather than retaining all possible tensors. This partial action approach achieves computational efficiency by retaining only the necessary data, avoiding excessive memory usage while still preventing unnecessary rematerialization operations.
Data Source
AI summary
A processor-implemented method includes receiving a graph representing an artificial neural network (ANN). The graph includes multiple nodes connected by edges and each node represents an operation. Retention intervals are determined for the multiple node outputs based on rematerialization constraints and paging constraints. The retention intervals correspond to a time interval for retaining each node output in at least one local memory. A sequence of tasks for executing the multiple nodes of the graph representing the ANN is determined based on the retention intervals.