Dictionary Compression Using Clustered Reference Data Units
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in constructing representative datasets for dictionary-based compression, especially in large data spaces, where standard compression algorithms like Zstd struggle to efficiently compress and decompress data due to the vast number of potential block values.
Innovation Solution
The method involves clustering data units into smaller clusters based on similarity, selecting reference units for each cluster, and using these reference units as dictionaries for compression, employing algorithms like PQk-means or LSH to reduce computational complexity and improve compression ratios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard compression algorithms like Zstd are used on large data spaces, then compression can be performed, but the computational complexity increases and compression ratios deteriorate due to the vast number of potential block values
Solution Approach 1:
The patent segments the large data space into multiple smaller clusters based on data similarity. Instead of performing compression across the entire large data space which creates high computational complexity, the system divides it into manageable clusters (e.g., using hash-based partitioning or k-means clustering). Each cluster is then compressed independently, significantly reducing the computational burden while maintaining effective compression ratios within each segment.
Solution Approach 2:
The patent applies different compression strategies or parameters to different clusters based on their local characteristics. Each cluster is treated as a distinct unit with its own compression dictionary or algorithm parameters, allowing the system to optimize compression locally for each data segment rather than using a uniform approach across the entire data space, thereby improving overall compression efficiency.
2Productivity
If clustering algorithms are used to reduce computational complexity, then compression speed improves, but the complexity of selecting reference units increases
Solution Approach 1:
The patent performs clustering and selects reference units in advance during an offline training phase, before the actual compression operation. By pre-processing the data to identify clusters and their representative reference units, the system eliminates the need for complex real-time clustering during compression. This preliminary action stores the cluster assignments and reference unit selections, which are then reused during compression to achieve high speed without algorithmic complexity.
Solution Approach 2:
The patent creates simplified representations of clusters by selecting representative reference units that capture the essential characteristics of each cluster. Instead of performing complex comparisons during compression, the system copies or references these pre-selected representative units, significantly simplifying the compression algorithm while maintaining effectiveness. The reference units serve as proxies for entire clusters, reducing computational requirements.
Data Source
AI summary
Data units of a dataset may be compressed by clustering the data units into clusters, selecting a reference unit for each unit cluster, and compressing data units of each unit cluster using the reference unit of the unit cluster as a dictionary. The computational efficiency of the clustering algorithm may be improved by not applying it to data units themselves, but rather to hash values of the data units, where the hash values have a much smaller size than the data units. The hash function may be a locality-sensitive hash (LSH) function. The reference unit of a cluster may be determined in any of a variety of ways, for example, by selecting a centroid or exemplar of the cluster. Clusters, including their references values, may be indexed in a cluster index (e.g., a Faiss index), which may be searched to assign future added or modified data units to clusters.