Nearest Neighbor Cluster Data Compression via Locality Sensitive Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in deduplication and compression due to the inability of cryptographic hash functions to identify nearest neighbors, leading to suboptimal data redundancy reduction and increased computational resources required for deduplication processes.
Innovation Solution
The use of locality sensitive hashing algorithms, such as k-means, k-medoids, mean shift, GMM, or DBSCAN, to create nearest neighbor clusters, allowing for improved deduplication and compression by grouping similar data blocks together, thereby reducing storage needs and computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If cryptographic hash functions are used for data deduplication, then data redundancy reduction is achieved, but search times increase and computational resources are consumed
Solution Approach 1:
The patent segments the data space into multiple clusters using k-means clustering, where each cluster contains data blocks with similar characteristics. This segmentation allows the search process to be divided into two stages: first identifying the relevant cluster, then searching only within that cluster, thereby reducing overall search time while maintaining deduplication effectiveness.
Solution Approach 2:
The patent performs preliminary clustering of data blocks using k-means algorithm before the actual deduplication search. By pre-organizing data into nearest-neighbor clusters based on similarity metrics, the system prepares the data structure in advance, enabling faster subsequent searches without compromising the accuracy of duplicate detection.
2Loss of substance
If traditional deduplication methods are used, then data redundancy is reduced, but storage requirements for tracking deduplicated data increase
Solution Approach 1:
The patent introduces a new dimensional organization of data by creating k-means clusters that group similar data blocks together. This additional dimensional structure allows the system to track deduplicated data more efficiently by referencing cluster identifiers and positions rather than maintaining extensive metadata for every individual data block, thereby reducing the storage footprint.
3Measurement precision
If exhaustive search methods are used for deduplication, then accurate duplicate identification is achieved, but processing speed decreases
Solution Approach 1:
The patent divides the exhaustive search space into multiple k-means clusters, each containing data blocks that are similar to each other. This segmentation transforms a single large exhaustive search into multiple smaller, targeted searches within clusters, maintaining duplicate identification accuracy while significantly improving processing speed by reducing the search space for each operation.
Solution Approach 2:
The patent applies local quality by ensuring that each k-means cluster contains data blocks with high local similarity. This local organization means that duplicate detection can be performed more efficiently within each cluster using optimized algorithms, while the overall system maintains global accuracy through the comprehensive cluster structure.
Data Source
AI summary
Disclosed are techniques for compressing data in a data storage system comprising searching a cluster of nearest neighbors, wherein the cluster has been created using a locality sensitive hashing algorithm, to determine if a data block can be compressed. In alternate embodiments, nearest neighbor clusters can be formed using unsupervised learning. Additionally, nearest neighbors can also be formed in alternate embodiments using one or more of the following algorithms: a k-means clustering algorithm, a k-medoids clustering algorithm, a mean shift algorithm, a generalized method of moment (GMM) algorithm, or a density based spatial clustering of applications with noise (DBSCAN) algorithm.