Predictive Data Placement Across Heterogeneous Storage Tiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Optimizing data placement in distributed storage systems with heterogeneous storage devices is challenging due to varying IO densities and deletion properties, requiring accurate predictions of data access patterns and lifetime to identify optimal storage devices that minimize cost and meet access requirements.
Innovation Solution
A predictive model is used to determine data temperature and lifetime, and a cost-benefit analysis selects the most suitable storage device based on these predictions, considering geographical location, network connectivity, and data erasure characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large data storage devices are used, then storage capacity is improved, but data access speed deteriorates
Solution Approach 1:
The patent segments the storage system into multiple heterogeneous storage devices with different characteristics (fast/small and slow/large). By dividing the storage workload across these segmented devices and using predictive modeling to route data appropriately, the system achieves both high capacity and fast access speeds simultaneously.
Solution Approach 2:
The patent applies local quality by assigning different storage devices with different performance characteristics to different data objects based on their predicted access patterns. Hot data is placed on fast storage devices while cold data is placed on large capacity devices, optimizing both access speed and storage capacity for different data segments.
2Ease of operation
If storage devices with fine granularity deletion properties are used, then data deletion flexibility is improved, but device complexity increases
Solution Approach 1:
The patent implements self-service by using predictive models that automatically analyze data object characteristics and determine optimal storage device selection. This automated decision-making process eliminates the need for manual intervention in deletion and placement operations, providing flexibility without increasing operational complexity.
Solution Approach 2:
The patent applies preliminary action by predicting data object characteristics (temperature, lifetime, access patterns) before data is stored. This advance prediction allows the system to pre-determine the optimal storage device and deletion strategy, simplifying subsequent operations while maintaining flexibility.
3Device complexity
If storage devices with large granularity deletion properties are used, then device complexity is reduced, but data deletion efficiency deteriorates
Solution Approach 1:
The patent changes the parameter of deletion granularity by dynamically selecting storage devices based on predicted data characteristics. The system can choose between fine-granularity and coarse-granularity deletion capabilities depending on the data object's predicted lifetime and access patterns, optimizing deletion efficiency without requiring all devices to have complex fine-granularity capabilities.
4Adaptability or versatility
If heterogeneous storage devices are used, then storage system versatility is improved, but data placement optimization difficulty increases
Solution Approach 1:
The patent implements feedback mechanisms where predictive models continuously analyze data object characteristics, access patterns, and storage device performance. This feedback loop enables the system to automatically adjust data placement decisions across heterogeneous devices, managing the complexity of versatile storage systems through automated monitoring and optimization.
Solution Approach 2:
The patent introduces predictive modeling and cost-benefit analysis as intermediary layers between the data placement request and the heterogeneous storage devices. This intermediary layer abstracts the complexity of managing diverse storage characteristics, translating high-level data requirements into optimal device selection without requiring direct complex management of each device's unique properties.
Data Source
AI summary
A method for optimizing data storage includes obtaining a data object for storage at memory hardware in communication with data processing hardware. The memory hardware includes a plurality of storage devices, each storage device of the plurality of storage devices including storage parameters different from each other storage device of the plurality of storage devices. The method also includes determining one or more data object parameters associated with the data object and predicting, using a model and the data object parameters and the storage parameters, an object temperature representative of a frequency of access for the data object and an object lifetime representative of an amount of time the data object is to be stored. The method further includes selecting, using the predicted object temperature and object lifetime, one of the storage devices, and storing the data object at the selected one of the storage devices.