Nearest Neighbor Clusters for Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in deduplication and compression due to the inability of cryptographic hash functions to identify nearest neighbors, leading to increased computational resources and potential hash collisions.
Innovation Solution
The use of locality sensitive hashing algorithms, such as k-means clustering, k-medoids, mean shift, GMM, or DBSCAN, to create nearest neighbor clusters, which allows for the identification of similar data blocks and reduces the search space for deduplication and compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If cryptographic hash functions are used for data deduplication, then data uniqueness can be determined, but computational resources increase and hash collisions occur
Solution Approach 1:
The patent segments the data deduplication process into two stages: first using locality-sensitive hashing to create hash values that group similar data blocks together, then using cryptographic hash functions only on the segmented results to determine final uniqueness. This segmentation reduces the overall computational burden while maintaining reliability.
Solution Approach 2:
The patent introduces locality-sensitive hashing as an intermediary mechanism between data blocks and cryptographic hash functions. This intermediary groups similar blocks before applying computationally intensive cryptographic hashing, thereby reducing total computational resources while preserving accurate uniqueness determination.
2Reliability
If cryptographic hash functions are used for data deduplication, then data uniqueness can be determined, but hash collisions increase
Solution Approach 1:
By segmenting the deduplication process into locality-sensitive hashing followed by cryptographic hashing, the patent reduces the probability of hash collisions. The first stage groups similar blocks, and the second stage resolves uniqueness within smaller subsets, thereby reducing collision opportunities while maintaining reliability.
Solution Approach 2:
Locality-sensitive hashing serves as an intermediary that pre-processes data blocks to group similar ones, reducing the likelihood that cryptographic hash functions will produce collisions. This intermediary layer filters and organizes data before the final uniqueness check, minimizing harmful collision effects.
3Loss of substance
If traditional deduplication methods are used, then data redundancy is reduced, but processing time increases
Solution Approach 1:
The patent applies locality-sensitive hashing as a preliminary action before performing full cryptographic hash comparisons. This preliminary grouping of similar data blocks significantly reduces the number of computationally intensive operations needed, thereby reducing processing time while still achieving effective redundancy reduction.
Solution Approach 2:
By dividing the deduplication process into segments (locality-sensitive hashing followed by cryptographic hashing), the patent reduces the time required for each stage. The segmentation allows parallel processing and reduces the overall computational path, thereby reducing processing time while maintaining redundancy reduction effectiveness.
Data Source
AI summary
Disclosed are techniques for data deduplication, which include methods, systems, or computer products for reducing data redundancy in a data storage system comprising searching a cluster of nearest neighbors, wherein the cluster has been created using a locality sensitive hashing algorithm, to determine if a data block has been stored in the data storage system prior to writing the data block. In alternate embodiments, the nearest neighbor clusters could be created using one or more of the following algorithms: k-means clustering algorithm, a k-medoids clustering algorithm, a mean shift algorithm, a generalized method of moment (GMM) algorithm, or a density based spatial clustering of applications with noise (DBSCAN) algorithm.