Similarity Based Compression for Data Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems face challenges in efficiently compressing vast amounts of data, leading to suboptimal utilization and increased storage needs.
Innovation Solution
The implementation of similarity-based compression methods, utilizing hashing techniques like locality sensitive hashing (LSH) to identify and compress redundant data blocks by calculating similarity hashes and storing delta representations, allowing for efficient data reduction by referencing similar data chunks already stored in the system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional compression methods are used, then storage capacity is improved, but compression ratio is insufficient and storage utilization remains suboptimal
Solution Approach 1:
The patent segments data into fixed-size chunks and processes them individually through hashing and similarity comparison. Each chunk is evaluated independently to identify redundant data, enabling fine-grained compression control and higher overall compression ratios while maintaining storage capacity.
Solution Approach 2:
The patent uses hashing to create compact representations (copies) of data chunks without storing the actual data. By comparing hash values, the system identifies redundant chunks and stores only unique data, achieving significant compression ratios while preserving storage capacity for essential information.
2Productivity
If data compression is increased, then storage utilization is improved, but data retrieval efficiency may be compromised
Solution Approach 1:
The patent introduces hash values as intermediary representations that enable efficient data retrieval. Instead of comparing entire compressed data blocks, the system compares compact hash values to identify redundant chunks, dramatically speeding up retrieval operations while maintaining high storage utilization through effective compression.
Solution Approach 2:
The patent replaces mechanical comparison of large data blocks with computational hash value comparison. This substitution transforms a computationally intensive process into a lightweight operation, maintaining fast retrieval speeds even as compression ratios increase and data volumes grow.
3Productivity
If similarity-based compression is implemented, then compression ratio is improved, but computational complexity increases
Solution Approach 1:
The patent extracts only the essential identifying feature of each data chunk (its hash value) for comparison purposes, rather than processing the entire chunk. This extraction approach dramatically reduces computational complexity while maintaining high compression ratios by accurately identifying redundant data through hash value matching.
Data Source
AI summary
A method for compressing an evaluated chunk (EC), the method includes (i) selecting a compressing scheme (SC) of a set of candidate SCs to provide a selected SC; and (ii) compressing the EC using the selected SC to provide a compressed EC. A similar chunk that is similar to the EC is tied to a reference chunk (RC) by having the similar chunk compressed based on the RC. The selecting includes evaluating candidate SCs of the set. The selecting is responsive to a first relationship between (i) a similarity score indicative of a similarity between the EV chunk and the similar chunk, and (ii) a similarity score indicative of a similarity between the similar chunk and the RC. A first candidate SC of the set includes untying the similar chunk from the RC, and tying the EV to the similar chunk. A second candidate SC of the set includes self-compressing the EV without tying the EV to the similar chunk.