Incremental Data Similarity via Frequency-Weighted Block Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for determining data similarity, such as de-duplication and Broder techniques, are computationally intensive and time-consuming, especially when dealing with large files, and lack accuracy in incremental similarity re-computation, leading to inefficiencies in data storage management.
Innovation Solution
The method involves calculating similarity based on a subset of frequency-weighted blocks from one dataset compared to another, allowing incremental updates without re-computing the entire dataset, and using block-filtering rules to enhance accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If brute force method is used to determine similarity between files, then complete accuracy is achieved, but computational complexity and time consumption increase significantly
Solution Approach 1:
The patent segments the file comparison process by dividing files into blocks and further into frequency-weighted subsets. Instead of comparing all blocks between two files, the system divides blocks into frequent and infrequent categories, and further segments frequent blocks into weighted subsets. This segmentation allows the similarity calculation to focus on the most significant blocks, reducing computational complexity while maintaining measurement accuracy.
Solution Approach 2:
The patent applies partial action by computing similarity based on a selected subset of frequency-weighted blocks rather than all blocks. The system identifies and processes only the most relevant blocks (those with higher frequencies and weights) to determine file similarity, achieving sufficient accuracy without the excessive computational effort of processing every single block.
2Device complexity
If Broder technique is used to limit comparison to k blocks, then computational complexity is reduced, but accuracy of similarity comparison deteriorates
Solution Approach 1:
The patent changes the parameters of block selection by introducing frequency weighting instead of simply limiting to k blocks. Each block is assigned a frequency weight based on its occurrence patterns, and the similarity calculation uses these weighted values. This parameter change allows the system to select blocks based on their significance rather than arbitrary count limits, improving accuracy while maintaining computational efficiency.
3Measurement precision
If re-computation of similarity is performed after data block changes, then accuracy is maintained, but time consumption increases due to re-sorting and re-hashing
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing frequency weights for all blocks before similarity comparison. When data blocks change, the system can update only the affected frequency weights and recalculate similarity using the pre-organized structure, rather than re-sorting and re-hashing all blocks from scratch. This preliminary organization enables faster incremental updates.
Solution Approach 2:
The patent introduces dynamics by enabling incremental updates of similarity metrics when data blocks change. Instead of static re-computation of all blocks, the system dynamically updates only the affected frequency-weighted subsets and adjusts similarity scores accordingly. This dynamic approach maintains accuracy while significantly reducing the time required for updates.
4Reliability
If all data blocks are processed for similarity determination, then complete data coverage is achieved, but storage management efficiency decreases
Solution Approach 1:
The patent applies local quality by treating different blocks differently based on their frequency characteristics. Frequent blocks receive higher weights and are processed with more attention, while infrequent blocks receive lower weights. This local differentiation allows the system to focus computational resources on the most significant blocks, improving storage management efficiency while maintaining reliable similarity determination.
Data Source
AI summary
Methods and systems for efficiently determining a similarity between two or more datasets. In one embodiment, the similarity is determined based on comparing a subset of sorted frequency-weighted blocks from one dataset to a subset of sorted frequency-weighed blocks from another dataset. Data blocks of a dataset are converted into hash values that are frequency-weighted. These frequency-weighted hash values can be compared to frequency-weighted hash values of another dataset to determine a similarity of the two datasets. In another embodiment, upon a change of a block in a subset of the dataset, the similarity value is re-determined without resorting or hashing the blocks of a dataset other than the blocks of the subset, resulting in an increased performance of a similarity comparison. In another embodiment, blocks of a dataset are excluded based on a block-filtering rule to increase the accuracy of the similarity comparison.