Active Data Set Forecasting for Dynamic Storage Cache Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems lack the ability to accurately forecast the amount of active data set, which is crucial for optimizing tiering and caching strategies, leading to inefficiencies in storage resource allocation and management.
Innovation Solution
A machine learning-based approach is employed to forecast the active data set by processing historical input/output requests, generating relevant features, and using a machine learning model to predict future activity levels, allowing for dynamic allocation of cache storage capacity, metadata, and resource balancing across storage nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional storage systems use static tiering and caching strategies, then system simplicity is maintained, but storage resource allocation efficiency deteriorates
Solution Approach 1:
The system performs preliminary actions by forecasting the active data set size for future time intervals before actual storage operations occur. The machine learning model predicts ADS size in advance, allowing the storage system to proactively allocate cache capacity and adjust tiering strategies before performance degradation occurs, rather than reacting to actual workload conditions
Solution Approach 2:
The patent replaces manual or rule-based mechanical decision-making for tiering and caching with an automated machine learning system. The ML model processes historical IO requests and automatically generates forecasts, substituting complex human-designed allocation rules with data-driven predictions that adapt dynamically to changing workload patterns
2Reliability
If storage systems allocate fixed cache capacity, then resource management simplicity is maintained, but cache hit ratio deteriorates
Solution Approach 1:
The system implements dynamic cache capacity allocation where the cache size is not fixed but continuously adjusted based on ML forecasts of the active data set. The cache capacity evolves over time to match predicted workload demands, transitioning from a static resource allocation model to a dynamic one that adapts to changing storage conditions
Solution Approach 2:
The system establishes a feedback loop where actual IO performance and cache effectiveness are continuously monitored, compared against ML predictions, and used to refine future cache allocation decisions. This closed-loop control ensures the cache hit ratio is maintained at optimal levels through continuous adjustment based on system performance feedback
3Measurement precision
If storage systems process all historical IO data in detail, then forecast accuracy is improved, but processing time and computational resources worsen
Solution Approach 1:
The system extracts only the most relevant features from historical IO requests for the machine learning model, rather than processing all raw IO data in detail. By identifying and extracting key characteristics that drive active data set changes, the system achieves high forecast accuracy while significantly reducing the computational burden of data processing
Solution Approach 2:
The system applies partial action by focusing computational resources on processing only the most critical historical data points and features that have the greatest impact on forecast accuracy. Rather than exhaustively analyzing every IO request, the ML model processes a curated subset of information that provides sufficient predictive power with minimal processing overhead
Data Source
AI summary
A method, computer program product, and computing system for processing a plurality of historical input/output (IO) requests associated with a storage object of a storage system. A plurality of IO features may be generated using the plurality of historical IO requests. An active data set for the storage object may be forecasted for a particular future time interval using a machine learning model based upon, at least in part, the plurality of IO features.