Context-Aware Memory Tiering for Deterministic ML Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional memory tiering solutions for DNN training are agnostic to deterministic access patterns, reactive, and reliant on semi-accurate telemetry, leading to inefficient memory usage and performance delays.
Innovation Solution
Implement context-aware memory tiering that proactively migrates data based on deterministic access patterns during DNN training, using hooks to collect context information and perform eviction and prefetching without reliance on telemetry data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional memory tiering solutions are used, then memory capacity is increased, but memory access performance deteriorates due to lack of awareness of deterministic access patterns
Solution Approach 1:
The system performs preliminary identification of deterministic access patterns during the forward pass, and proactively migrates data between memory tiers before the backward pass begins. This preliminary action ensures that frequently accessed data is already in optimal memory locations when needed, eliminating performance penalties during actual computation.
Solution Approach 2:
The system uses hooks to collect context information about data access patterns during training, creating a feedback mechanism that identifies deterministic access patterns. This feedback enables the system to adaptively manage memory tiering decisions, improving access performance while maintaining expanded memory capacity.
2Extent of automation
If telemetry-based memory tiering is used, then memory management is automated, but accuracy deteriorates due to semi-accurate telemetry data
Solution Approach 1:
The system extracts context information directly from the training computation flow using hooks, separating the accuracy-critical access pattern detection from the potentially inaccurate telemetry data. By taking out the essential context information needed for precise pattern identification, the system achieves both automation and high measurement precision.
3Quantity of substance
If reactive memory tiering is used, then memory usage is optimized, but training time increases due to delays in detecting hot pages
Solution Approach 1:
The system performs memory tiering decisions during the forward pass as a preliminary action, completing data migration before the backward pass begins. This eliminates the time delay associated with reactive detection during computation, as all optimization actions are already in place before the time-critical backward pass starts.
Solution Approach 2:
The system leverages the periodic nature of DNN training workloads, where the same data is accessed in deterministic patterns during alternating forward and backward passes. By synchronizing memory tiering actions with this periodic access pattern, the system optimizes memory usage without adding time overhead to the training cycle.
Data Source
AI summary
Techniques for training machine learning models are described. In particular, some examples describe the use of storing out a tensor after a training forward pass if conditions warrant this storage. For example, if the tensor can be stored to a different memory, but still be pre-fetched before it is needed in a backward training pass, then the tensor is stored out in some examples. By storing out tensors, memory is freed for computation of subsequent forward and backward passes. This helps improve page swapping, etc. of data.