Data Placement by Temperature and Lifetime Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Optimizing data placement in distributed storage systems with heterogeneous storage devices is challenging due to varying IO densities and deletion properties, requiring accurate predictions of data records' access patterns and lifetimes to minimize storage costs.
Innovation Solution
A predictive model is used to determine data temperature and lifetime, allowing for the selection of optimal storage devices based on these parameters, incorporating cost-benefit analysis to minimize storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large data storage devices are used, then storage capacity is improved, but data access frequency deteriorates
Solution Approach 1:
The system segments storage devices into different types (large capacity devices and small capacity devices) and assigns data objects to appropriate segments based on their temperature and lifetime characteristics. Hot data with short lifetime is placed in small capacity devices for fast access, while cold data with long lifetime is placed in large capacity devices for cost-effective storage.
Solution Approach 2:
Different storage devices are assigned different local qualities (performance characteristics) based on data requirements. Small capacity devices provide high access speed for frequently accessed data, while large capacity devices provide cost-effective storage for infrequently accessed data. The system dynamically assigns data to devices with appropriate local quality based on predicted temperature and lifetime.
2Ease of operation
If storage devices with fine granularity deletion properties are used, then data deletion flexibility is improved, but device complexity increases
Solution Approach 1:
The system implements self-service deletion by predicting data lifetime and automatically deleting data objects when their predicted lifetime expires. The data object parameters include deletion triggers, and the system autonomously manages deletion without requiring manual intervention or complex deletion scheduling mechanisms.
3Device complexity
If storage devices with large granularity deletion properties are used, then device complexity is reduced, but data deletion efficiency deteriorates
Solution Approach 1:
The system performs preliminary action by predicting data lifetime before deletion is needed. By estimating how long data will be retained based on temperature and lifetime predictions, the system proactively manages deletion timing and triggers deletion when appropriate, avoiding the inefficiency of bulk deletion operations.
4Productivity
If accurate predictions of data properties are made, then data placement optimization is improved, but measurement precision requirements increase
Solution Approach 1:
The system implements feedback by continuously monitoring actual data access patterns and comparing them with predicted temperature and lifetime values. This feedback mechanism allows the system to refine its predictions over time and adjust data placement decisions based on actual observed behavior rather than relying solely on initial predictions.
Data Source
AI summary
A method for optimizing data storage includes obtaining a data object for storage at memory hardware in communication with data processing hardware. The memory hardware includes a plurality of storage devices, each storage device of the plurality of storage devices including storage parameters different from each other storage device of the plurality of storage devices. The method also includes determining one or more data object parameters associated with the data object and predicting, using a model and the data object parameters and the storage parameters, an object temperature representative of a frequency of access for the data object and an object lifetime representative of an amount of time the data object is to be stored. The method further includes selecting, using the predicted object temperature and object lifetime, one of the storage devices, and storing the data object at the selected one of the storage devices.