ML Skew Prediction for Multi-Tier Storage Data Movement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems with multiple tiers face inefficiencies in data movement due to the lack of accurate prediction of IO activity patterns, leading to suboptimal distribution of data across tiers and inefficient migration processes.
Innovation Solution
Implementing a machine learning-based skew prediction system that processes IO activity data across different time granularities to generate predictive models, allowing for the conversion of skew measurements and guiding data movement decisions between storage tiers, thereby improving data placement and migration strategies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data movement decisions are based on actual skew measurements across multiple time granularities, then data placement accuracy improves, but computational complexity and measurement time increase
Solution Approach 1:
The system pre-generates predictive models during off-peak periods that can quickly estimate skew measurements for any time granularity. When data movement decisions are needed, these pre-computed models provide immediate predictions without requiring real-time calculation across multiple time granularities, thus maintaining measurement accuracy while reducing decision-making time.
Solution Approach 2:
The predictive model creates virtual representations (copies) of skew measurement data across different time granularities. Instead of performing actual measurements at multiple time scales, the system generates predictive copies of what the measurements would be, allowing rapid data movement decisions based on these simulated measurements without the computational overhead of actual multi-granularity measurements.
2Productivity
If comprehensive IO activity analysis across multiple time granularities is performed, then data distribution optimization improves, but computational resources required increase
Solution Approach 1:
The predictive model allows the system to change the time granularity parameter dynamically based on needs. Instead of always performing comprehensive multi-granularity analysis, the system can select appropriate time granularities using the predictive model, reducing computational energy while maintaining effective data distribution optimization.
Solution Approach 2:
The predictive model serves multiple functions: it can predict skew measurements for any time granularity, guide data movement decisions, and optimize data distribution. This single multi-functional component replaces what would otherwise require separate comprehensive analysis systems for each time granularity, reducing overall computational energy requirements.
3Measurement precision
If real-time skew measurements are used for data movement decisions, then data placement accuracy improves, but system complexity increases
Solution Approach 1:
The predictive model acts as an intermediary between raw IO activity data and data movement decisions. Instead of directly implementing complex real-time measurement systems across multiple time granularities, the predictive model mediates by providing simplified predictions that capture the essential skew information needed for accurate data placement without the full complexity of real-time multi-granularity measurements.
4Reliability
If multiple time granularities are used for skew measurement, then prediction accuracy improves, but data movement processing time increases
Solution Approach 1:
The system performs the computationally intensive work of analyzing multiple time granularities in advance to build the predictive model. During actual data movement operations, the pre-computed model provides rapid predictions that maintain high accuracy without the processing delay of real-time multi-granularity analysis, thus improving data movement speed while preserving prediction reliability.
Data Source
AI summary
An apparatus comprises a plurality of storage tiers, at least one data mover module, and a machine learning system coupled to the data mover module. The machine learning system comprises a model generator and a skew predictor. The model generator processes information characterizing input-output activity involving one or more of the storage tiers in order to obtain skew measurements in different time granularities, with the skew measurements indicating portions of the input-output activity directed to portions of the storage tier(s), and generates a predictive model from the skew measurements. The skew predictor is configured in accordance with the predictive model to convert skew measurements in one of the time granularities to corresponding skew measurements in another one of the time granularities. One or more of the converted skew measurements are utilized by the data mover module in controlling transfer of data between the storage tiers.