Data-Aware Storage Tiering for Deep Learning IO Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Network (DNN) training with large datasets faces significant IO bottlenecks due to inefficient use of multi-tier storage systems, where conventional training methods reload data from a capacity tier for each epoch, leading to stalls and suboptimal performance.
Innovation Solution
The proposed solution involves splitting an epoch into multiple mini-epochs and prefetching the next mini-epoch while training, using a repeating factor to optimize IO bandwidth and model convergence, with adaptive adjustments based on model convergence feedback to minimize stalls and maintain training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If training data is loaded randomly from the entire dataset on every training iteration, then model convergence is achieved, but IO stalls occur due to data not being cached in the performance tier
Solution Approach 1:
The system performs preliminary actions by prefetching future training data into the performance tier storage before it is actually needed for training. The data manager identifies and loads data that will be required in upcoming training iterations, eliminating IO stalls by ensuring data is already cached and ready when needed.
Solution Approach 2:
The training dataset is segmented into different portions stored across multiple storage tiers (performance tier and capacity tier). The data manager divides data access into frequent accesses from the performance tier for active training and less frequent accesses from the capacity tier, organizing data segments based on their access patterns and requirements.
2Quantity of substance
If the entire dataset is maintained on the capacity tier with large capacity, then storage capacity requirements are met, but read throughput is slow causing training inefficiency
Solution Approach 1:
Different storage tiers are assigned different quality characteristics based on local requirements. The performance tier provides high-speed read access for actively training data, while the capacity tier provides large-capacity storage for the entire dataset. Each tier is optimized for its specific function, with the data manager intelligently distributing data across tiers based on access patterns.
Solution Approach 2:
The performance tier acts as an intermediary between the capacity tier and the training system. It buffers and caches data from the capacity tier, providing high-speed access to frequently used training data while the capacity tier maintains the complete dataset. This intermediary layer resolves the conflict between capacity and speed by decoupling storage from access.
3Adaptability or versatility
If conventional training loads new training data from the entire dataset on every iteration, then data diversity is maintained, but IO bandwidth is not optimally utilized
Solution Approach 1:
The data manager implements feedback mechanisms by monitoring training progress, data access patterns, and storage system performance. Based on this feedback, it dynamically adjusts data prefetching strategies, identifies which data will be needed next, and optimizes IO bandwidth utilization. The system learns from past access patterns to improve future data loading efficiency while maintaining data diversity.
Data Source
AI summary
Systems and methods are configured to split an epoch associated with a training dataset into a plurality of mini-epochs. A machine learning model can be trained with a mini-epoch of the plurality of mini-epochs. The mini-epoch can be, during the training, iterated for a number of times during the training. One or more metrics reflective of at least one of: a training loss, training accuracy, or validation accuracy of the machine learning model associated with the mini-epoch can be received. Whether to terminate iterations of the mini-epoch early before a number of iterations of the mini-epoch reaches the number of times based on the one or more metrics can be determined. The number of iterations can be a non-zero number.