Document Fingerprinting Sketches Using One-Bit Fingerprints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing algorithms for identifying duplicate and near-duplicate documents on the Internet require large storage space, making them inefficient for handling millions of documents.
Innovation Solution
A method that creates sketches of documents using one-bit fingerprints, reducing storage space requirements while maintaining accuracy in identifying and grouping duplicate and near-duplicate documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional algorithms store multiple full-precision fingerprints (e.g., 64-bit integers) as sketches, then document similarity can be determined accurately, but storage space requirements become very large
Solution Approach 1:
The patent segments each fingerprint into multiple portions (e.g., four 16-bit portions from a 64-bit fingerprint) and stores only selected segments. This segmentation allows reducing storage from full-precision fingerprints to partial fingerprints while maintaining sufficient accuracy for similarity determination through probabilistic matching of the stored segments.
Solution Approach 2:
The patent uses simplified, lower-precision fingerprint representations (partial fingerprints) as disposable approximations of full fingerprints. These partial fingerprints require less storage space and can be regenerated or replaced, providing a cost-effective solution for large-scale document fingerprinting while maintaining acceptable accuracy through statistical properties.
2Reliability
If more sketches are stored for each document, then duplicate document identification accuracy improves, but storage requirements increase significantly
Solution Approach 1:
The patent segments fingerprints and stores multiple sketches of partial fingerprints instead of full fingerprints. This allows creating multiple sketches (e.g., 6 or more) with reduced storage per sketch, maintaining reliability through the law of large numbers where multiple partial matches provide statistically significant similarity determination.
Solution Approach 2:
The patent changes the precision parameter of fingerprints from full-precision (64-bit) to reduced-precision (partial 16-bit segments). This parameter change enables storing multiple sketches within the same storage budget, improving duplicate identification reliability through increased sample size while accepting controlled precision reduction.
3Loss of information
If full-precision fingerprints are used for all documents, then fingerprint uniqueness is maximized, but processing and storage efficiency decrease
Solution Approach 1:
The patent extracts only specific portions (segments) from full fingerprints for storage and comparison. By taking out and storing only the necessary segments rather than complete fingerprints, the system achieves processing efficiency gains while retaining sufficient information for duplicate detection through selective segment matching.
Solution Approach 2:
The patent uses simplified partial fingerprints as efficient proxies for full fingerprints. These cheaper, reduced-precision representations enable faster processing and storage operations, with the understanding that some information is sacrificed but sufficient accuracy remains for the intended application of duplicate document identification.
Data Source
AI summary
Methods and computer program products for creating sketches of a document, which are compared with sketches of other documents, in order to determine the documents' degree of similarity. A sketch is a digest of information from random locations within a document. A document is divided into a set of shingles. Each shingle is converted into a set of fingerprints. A sketch is determined based on one bit fingerprints thus created. In order to create additional sketches of the document, a new set of fingerprints are created by randomization techniques.