Entropy-Guided Data-Aware Compression for Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems lack efficiency in compressing data due to the use of general-purpose compression methods, which do not account for specific data types like time-series or image data, leading to suboptimal compression ratios.
Innovation Solution
A data-aware compression system that selects a suitable compression scheme based on entropy levels of pre-compressed data, applying type-based compression schemes for time-series or image data, and general-purpose compression when type-specific schemes are inadequate, with adaptive entropy thresholds for improved compression ratios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If general-purpose compression is used on all data, then the system can handle various data types, but the compression ratio is suboptimal for specific data types
Solution Approach 1:
The patent segments the compression process into multiple stages: first applying a generic compression algorithm to all data, then analyzing the compressed output to detect data type characteristics, and finally applying specialized compression algorithms for specific data types like time-series or images. This segmentation allows the system to maintain versatility while achieving optimal compression for each data type.
Solution Approach 2:
The patent performs preliminary compression using a general-purpose algorithm before detecting the data type. This preliminary action reduces the data size initially and creates a basis for subsequent type-specific compression, ensuring that even if type detection fails, the data has already been compressed to some extent.
2Loss of substance
If type-specific compression is applied, then compression ratio improves for specific data types, but the system complexity increases
Solution Approach 1:
The patent implements a dynamic compression system that adapts its approach based on the detected data type. The system maintains multiple compression algorithms and dynamically selects the appropriate one after type detection, allowing complexity to be introduced only when beneficial for specific data types while keeping the base system simple.
Solution Approach 2:
The patent introduces an intermediary data type detection mechanism that analyzes the compressed data to identify patterns characteristic of specific data types like time-series or images. This intermediary layer acts as a bridge between the generic compression input and the specialized compression algorithms, adding intelligence without requiring the entire system to be complex.
3Loss of substance
If multiple compression algorithms are maintained, then optimal compression can be achieved for different data types, but the processing time increases
Solution Approach 1:
The patent applies partial action by using a lightweight general-purpose compression algorithm on all data first, then applying more intensive type-specific algorithms only to the portion of data that benefits from them. This approach avoids the excessive processing time that would result from applying all algorithms to all data, while still achieving optimal compression where needed.
Solution Approach 2:
The patent performs preliminary compression with a fast generic algorithm before applying slower, more specialized algorithms. This preliminary action ensures that even if type detection or subsequent specialized compression is skipped, the data has already undergone some compression, minimizing the time penalty for maintaining multiple algorithms.
Data Source
AI summary
A method for storage system data aware compression, the method may include pre-compressing data units received by the storage system, by different pre-compression units to provide different pre-compressed versions of the data units; wherein the different pre-compression schemes are associated with different compression schemes, wherein at least some of the different compression schemes are data type specific compression schemes; calculating entropies of the different pre-compressed versions; and selecting a compression scheme out of the different compression schemes based on the entropies of the different pre-compressed versions.