Compression Sampling for Tiered Storage Data Steering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems face inefficiencies in managing data across multi-tiered storage systems, particularly in determining optimal storage locations based on data compressibility, leading to suboptimal performance, reliability, and power consumption.
Innovation Solution
The method involves ranking storage areas based on properties such as latency and durability, using compression sampling to identify data compressibility, and dynamically steering data between tiers based on heat detection and compression algorithm analysis, optimizing storage by distinguishing metadata from content and adapting algorithms based on file system types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in multi-tiered storage systems without compression sampling, then storage capacity is maintained, but performance and power consumption are suboptimal
Solution Approach 1:
The system performs compression sampling on data before final storage placement to predict compressibility characteristics. This preliminary analysis allows the storage system to pre-determine optimal storage locations and compression strategies, avoiding the need for continuous monitoring and adjustment during operation, thereby improving performance while reducing power consumption.
Solution Approach 2:
The system dynamically changes storage parameters such as compression ratio and storage tier selection based on the compressibility score obtained from sampling. By adjusting these parameters according to actual data characteristics rather than using fixed settings, the system optimizes both performance and energy efficiency for different data types.
2Reliability
If compression sampling is performed on all data, then optimal storage placement is achieved, but processing time and system complexity increase
Solution Approach 1:
Instead of analyzing all data, the system creates and analyzes a small sample copy of the data to determine compressibility characteristics. This sampling approach provides sufficient information for optimal storage placement decisions without requiring complex processing of the entire dataset, thus reducing system complexity while maintaining reliability.
Solution Approach 2:
The system performs compression sampling at a partial level rather than complete analysis of all data. By sampling only the necessary portion of data to determine compressibility, the system achieves adequate optimization without the excessive processing time and complexity that would result from full data analysis.
3Measurement precision
If multiple compression algorithms are tested on data samples, then compression savings accuracy is improved, but processing time increases
Solution Approach 1:
The system tests multiple compression algorithms on data samples in advance to determine which algorithms provide the best compression savings for different data types. These results are stored and reused for future storage decisions, so the time-consuming multiple algorithm testing is performed only once per data type rather than repeatedly, improving accuracy without continuous time loss.
Solution Approach 2:
The system applies multiple compression algorithms only to samples rather than complete datasets. This partial application provides sufficiently accurate compression savings predictions for storage placement decisions while minimizing the processing time penalty associated with testing multiple algorithms.
Data Source
AI summary
A method for data steering in tiered storage is described. In one embodiment, the method includes ranking a plurality of storage areas of a storage device according to at least one property of the plurality of storage areas. In some embodiments, the plurality of storage areas include at least a first storage area and a second storage area. The method includes obtaining a sample of data at the storage device, passing the sample of data through one or more compression codecs, and analyzing a result of passing the sample of data through the one or more compression codecs. In some embodiments, the result includes a score of compression savings associated with the sample of data. The method includes storing the data in one of the plurality of storage areas based at least in part on the analyzing the result of passing the sample of data through the one or more compression codecs.