Storage Object Grouping for ML Forecasting Resource Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine-learning based performance forecasting for storage systems is computationally expensive due to high memory footprint and CPU overhead, making it prohibitive for systems with millions of storage objects, despite offering better accuracy than statistical methods.
Innovation Solution
Divide storage objects into groups based on past activity levels, using a high-cost machine learning model for active objects, a low-cost machine learning model for intermediate objects, and a statistical method for dormant objects, thereby optimizing resource usage while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models are used to forecast IO performance for all storage objects, then forecasting accuracy is improved, but memory footprint and CPU overhead become prohibitive
Solution Approach 1:
The patent segments storage objects into multiple groups (hot, warm, cold) based on their activity patterns. Different machine learning models are applied to different segments: complex models for hot storage objects where accuracy is critical, and simpler models for warm/cold objects where computational resources are limited. This segmentation resolves the contradiction by applying computational intensity selectively rather than uniformly across all storage objects.
Solution Approach 2:
The patent applies different levels of modeling complexity to different parts of the storage system based on local characteristics. Hot storage objects receive high-accuracy machine learning forecasts, while warm and cold objects receive simplified forecasts. This local quality approach optimizes the trade-off between accuracy and computational cost by matching model complexity to the actual needs of each storage object group.
2Measurement precision
If machine learning models are applied to all storage objects uniformly, then forecasting precision is improved, but system complexity and resource requirements increase significantly
Solution Approach 1:
The patent divides storage objects into distinct segments (hot, warm, cold groups) and applies different modeling strategies to each segment. This segmentation reduces overall system complexity by avoiding the application of complex machine learning models to all objects, instead using simplified models for segments where high precision is not critical.
Solution Approach 2:
The patent applies machine learning models partially - only to the extent necessary for each storage object group. For hot objects, full machine learning models are applied; for warm and cold objects, simplified or no models are used. This partial action approach achieves sufficient precision where needed while avoiding excessive complexity elsewhere.
3Measurement precision
If complex machine learning models are used for all storage objects, then forecast accuracy is improved, but memory footprint increases prohibitively
Solution Approach 1:
The patent segments storage objects by activity level and applies different model complexities accordingly. Complex machine learning models are deployed only for hot storage objects that require high accuracy, while simpler models are used for warm and cold objects. This segmentation dramatically reduces the overall memory footprint compared to applying complex models to all objects uniformly.
Solution Approach 2:
The patent implements local quality by deploying high-complexity models only where locally necessary - for hot storage objects with high access patterns. For warm and cold objects, the system uses memory-efficient simplified models. This localized approach to model complexity achieves the required forecast accuracy for critical objects while minimizing memory consumption system-wide.
Data Source
AI summary
A method, computer program product, and computing system for determining a respective past activity level associated with a plurality of storage objects. The plurality of storage objects are divided into a plurality of storage object groups based upon, at least in part, the respective past activity level associated with the plurality of storage objects. Input/output (IO) performance data for a first storage object group of the plurality of storage object groups is forecasted using a first machine learning model. IO performance data for a second storage object group of the plurality of storage object groups is forecasted using a statistical method.