Multi-Tier Storage Compression for Cost-Latency Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-tiered storage systems face challenges in determining which data to store in each tier while maintaining low latency and optimizing storage costs, as they often fail to consider both latency and access patterns, and typically use a single compression technique or neglect to optimize latency.
Innovation Solution
A storage system that includes a dataset extractor module, data partitioner module, compression predictor module, and optimization engine to dynamically partition datasets, assign priorities based on access patterns, and select optimal compression schemes and storage tiers for each data partition, minimizing costs and maintaining low latency by using a cost function that considers economic cost, latency, and compression performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in low-latency storage tiers, then read/write latency is reduced, but economic cost increases
Solution Approach 1:
The storage system divides data into multiple data partitions and applies different storage strategies to each partition based on its access characteristics. High-priority partitions are stored in low-latency tiers while low-priority partitions are stored in high-latency, low-cost tiers, thus resolving the contradiction between latency and cost.
Solution Approach 2:
Different storage tiers are assigned to different data partitions based on their specific access requirements. Each data partition receives a customized storage location and compression scheme tailored to its access patterns, optimizing the balance between latency and cost for each local segment of data.
2Quantity of substance
If compression is applied to reduce storage costs, then storage capacity is optimized, but computing cost for compression and decompression increases
Solution Approach 1:
The system dynamically selects compression schemes based on data characteristics and access patterns. Different compression algorithms and compression ratios are applied to different data partitions, optimizing the balance between storage capacity utilization and computing overhead for each specific data set.
3Quantity of substance
If multiple compression schemes are supported to optimize storage, then storage efficiency is improved, but device complexity increases
Solution Approach 1:
The system dynamically selects and switches between different compression schemes based on real-time data characteristics and access patterns. The compression scheme for each data partition is not fixed but can be adjusted and optimized over time, managing complexity through adaptive rather than static multi-scheme support.
Data Source
AI summary
Some techniques described herein relate to determining how to optimally store datasets in a multi-tiered storage device with compression. In one example, a method includes assigning, to a data partition of a dataset, a priority based on access patterns of the data partition. Compression data is accessed describing results of compressing a data sample associated with the data partition using multiple compression schemes. Based both on the priority of the data partition and the compression data, a storage tier is determined for storing the data partition in the multi-tiered storage device. Further, based both on the priority of the data partition and the compression data, a compression scheme is determined for compressing the data partition for storage in the multi-tiered storage device. The data partition is compressed using the compression scheme to produce a compressed data partition, and the compressed data partition is stored in the storage tier.