Document Sketching via Sentence Delimiters and Hash Permutations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document similarity measurement methods are inefficient due to content overlap-based approaches, which do not allow for computationally efficient similarity computations, especially when dealing with documents of varying sizes and formats, and fail to effectively handle partial matches.
Innovation Solution
A method that computes a unique and compact document sketch using sentence-based logical delimiters, extracting significant words by their weight, and computing hash permutations, allowing for efficient similarity and containment checks, and adaptable clustering based on document distance metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If content overlap-based approaches are used to measure document similarity, then measurement precision is improved, but productivity deteriorates due to computational inefficiency
Solution Approach 1:
The patent extracts a compact sketch or fingerprint from each document that captures essential content characteristics. This sketch is a condensed representation that can be computed efficiently and used for rapid similarity comparison, eliminating the need to process entire documents while preserving measurement precision.
Solution Approach 2:
The patent transforms the document representation from full content to a condensed sketch with specific parameters (such as term frequencies, document length, and content density). This parameter transformation enables efficient computation while maintaining the ability to measure similarity accurately.
2Measurement precision
If full document comparison is performed, then measurement precision is improved, but loss of time increases due to processing large documents
Solution Approach 1:
The patent extracts a compact sketch from each document that contains essential information for similarity measurement. This sketch is significantly smaller than the original document but retains the key characteristics needed for accurate comparison, thereby reducing computation time while maintaining precision.
Solution Approach 2:
The patent performs preliminary processing to create document sketches before actual similarity comparisons are needed. These pre-computed sketches can be stored and reused for multiple comparisons, eliminating the need to reprocess full documents each time a similarity check is required.
3Productivity
If document sketches are made smaller for efficient computation, then productivity is improved, but loss of information increases
Solution Approach 1:
The patent transforms document content into a sketch with carefully selected parameters (term frequencies, document length, content density) that preserve essential information. This parameter transformation maintains the ability to measure similarity accurately while achieving compact representation for efficient computation.
Solution Approach 2:
The patent applies different processing qualities to different parts of the document. Important terms and phrases are preserved with higher fidelity in the sketch, while less critical content is summarized or omitted. This local quality approach ensures that the most informationally valuable aspects are retained in the compact representation.
4Reliability
If traditional fingerprinting algorithms are used, then uniqueness is improved, but adaptability deteriorates for varying degrees of similarity
Solution Approach 1:
The patent implements a dynamic similarity measurement system that can adapt to varying degrees of partial matches. The sketch-based approach allows for flexible comparison thresholds and can handle documents that overlap to varying degrees, unlike rigid traditional fingerprinting algorithms that are optimized only for near-exact matches.
Data Source
AI summary
A first embodiment of the invention provides a system that automatically classifies documents in a collection into clusters based on the similarities between documents, that automatically classifies new documents into the right clusters, and that may change the number or parameters of clusters under various circumstances. A second embodiment of the invention provides a technique for comparing two documents, in which a fingerprint or sketch of each document is computed. In particular, this embodiment of the invention uses a specific algorithm to compute the document's fingerprint. One embodiment uses a sentence in the document as a logical delimiter or window from which significant words are extracted and, thereafter, a hash is computed of all pair-wise permutations. Words are extracted based on their weight in the document, which can be computed using measures such as term frequency and the inverse document frequency.