Bloom Filter Document Similarity Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional document comparison methods are inefficient as they require direct access to documents, are time-consuming, and sensitive to minimal changes, and struggle to determine similarity between documents beyond identity.
Innovation Solution
The use of Bloom filters, which are probabilistic data structures that represent sets of hash values, allowing for efficient comparison and identification of near-duplicate or related documents by breaking down data into chunks, normalizing data, and computing hash values to construct and compare Bloom filters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If character-by-character comparison is used to determine document identity, then measurement precision is improved, but loss of time increases and device complexity increases
Solution Approach 1:
The patent segments documents into fixed-size chunks or blocks, transforming the comparison task from character-by-character to chunk-level operations. This segmentation enables parallel processing and reduces the time complexity from O(n) to O(n/k) where k is the chunk size, while maintaining comparison accuracy through hash-based representation of each chunk.
Solution Approach 2:
The patent applies preliminary hashing to document chunks before comparison, pre-computing hash values that can be stored and reused. This preliminary action transforms the original documents into compact hash representations, enabling rapid subsequent comparisons without re-processing the entire document content each time.
2Measurement precision
If character-by-character comparison is used to ensure exact document identity, then measurement precision is improved, but productivity deteriorates
Solution Approach 1:
The patent creates hash copies of document chunks as surrogate representations for comparison. Instead of comparing original document content directly, the system compares compact hash values that serve as faithful representations, dramatically increasing processing throughput while maintaining the ability to detect exact and near-duplicates.
Solution Approach 2:
The patent changes the parameter of comparison from raw document content to hash values of fixed length. This parameter transformation converts an O(n) comparison operation into an O(1) operation relative to document size, since hash values have fixed length regardless of the original document size, thereby improving productivity.
3Loss of time
If one-way function values are used to represent documents for comparison, then loss of time is reduced and device complexity is reduced, but loss of information increases
Solution Approach 1:
The patent uses partial hashing by computing hashes of fixed-size chunks rather than entire documents, and uses excessive hashing by applying multiple hash functions (including cryptographic and non-cryptographic variants) to the same data. This partial yet excessive approach maintains sufficient information for comparison purposes while achieving time and space efficiency.
Solution Approach 2:
The patent employs composite hashing by combining multiple different hash functions (cryptographic hashes like SHA-256 and non-cryptographic hashes like MurmurHash or FNV) to create a composite representation. This composite approach compensates for information loss in individual hash functions by leveraging the strengths of different hashing algorithms, maintaining both efficiency and informational fidelity.
4Reliability
If cryptographic hash functions are used to prevent accidental collisions, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent merges multiple hash functions of different types (cryptographic and non-cryptographic) into a composite hashing system. By combining the high collision resistance of cryptographic hashes with the computational efficiency of non-cryptographic hashes, the system achieves reliable collision prevention without the full complexity overhead of using only cryptographic functions for all operations.
Data Source
AI summary
A method for estimating similarity between two collections of information is described herein. The method includes comparing a first Bloom filter representing a first collection of information and a second Bloom filter representing a second collection of information, and determining a measure of similarity between the first collection of information and the second collection of information based on the comparing.