Customizable Progressive Data-Tiering Service for Storage Cost and Latency Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed systems face inefficiencies in data storage due to less frequently accessed data being stored in high-performance resources, leading to increased costs and latency issues when relocated to lower-cost storage, which can negatively impact client operations.
Innovation Solution
Implementing a data tiering service with a warm tier, multiple intermediate tiers, and a cold tier, where data items are dynamically promoted or demoted based on client-defined service level agreements (SLAs) and latency targets, allowing for smooth transitions in access latency and efficient resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is relocated from high-performance storage to lower-cost storage, then storage costs are reduced, but access latency increases significantly
Solution Approach 1:
The storage system is divided into multiple tiers (hot, warm, cold) with progressively lower performance and cost characteristics. Data is segmented across these tiers based on access patterns, allowing the system to optimize both cost and latency by keeping frequently accessed data in high-performance storage while moving infrequently accessed data to lower-cost storage.
Solution Approach 2:
The system dynamically adjusts data placement across storage tiers based on changing access patterns and service level agreements. Data can be promoted or demoted between tiers as access requirements change, allowing the system to adapt to varying performance needs while optimizing storage costs in real-time.
2Loss of time
If data is stored in high-performance resources, then access latency is reduced, but storage costs increase
Solution Approach 1:
Different portions of the storage system are assigned different quality characteristics based on local needs. High-performance storage resources are allocated to data requiring low latency access, while lower-cost resources handle data with less stringent access requirements. This localized optimization allows the system to minimize overall costs while meeting specific performance requirements for critical data.
Solution Approach 2:
The system changes the performance parameters of storage resources by adjusting data placement across tiers with different performance characteristics. By modifying which data resides in which storage tier based on access patterns and SLAs, the system dynamically optimizes the balance between access latency and storage costs without requiring hardware changes.
3Productivity
If data is relocated to lower-cost storage, then storage resource efficiency is improved, but client operation performance is negatively impacted
Solution Approach 1:
The system continuously monitors data access patterns and client performance requirements, using this feedback to dynamically adjust data placement decisions. When access patterns indicate that data should be promoted to higher-performance tiers to meet SLAs, the system automatically makes these adjustments, ensuring client performance requirements are maintained while optimizing storage resource efficiency.
Solution Approach 2:
The system proactively manages data placement by predicting future access patterns and pre-positioning data in appropriate storage tiers before access is needed. This preliminary action allows the system to maintain optimal performance while efficiently utilizing storage resources, as data is already in the correct tier when access patterns indicate it will be needed.
Data Source
AI summary
A tiering service enables a client to custom specify service level agreements for data items to be tiered and automatically promotes and demotes the data items amongst a warm tier, a plurality of intermediate tiers, and a cold tier to ensure the service level agreement commitments are met. In some embodiments, a client specifies segmentation criteria for defining multiple segments of data items included in a data scope or table and assigns latency targets to the segments in order to define the service level agreement. Also, in some embodiments, a plurality of intermediate tiers are implemented on common underlying hardware by varying metadata management to implement intermediate tiers that have progressively increasing latencies.