Statistical Data Fingerprinting for Document Similarity Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data loss prevention methods struggle to accurately categorize and compare portions of text within documents, as trivial changes can evade detection through simple hashing methods.
Innovation Solution
A computer-implemented method of statistical data fingerprinting that generates fingerprints by applying a statistical function measuring randomness of characters in subsets of text against an expected distribution, allowing for precise matching between document subsets using Euclidean, rectilinear, or Shannon entropy metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If simple block-hashing is used to compare documents or portions of documents, then the process is computationally efficient and simple, but trivial changes to the data can evade detection
Solution Approach 1:
The patent divides documents into multiple portions or segments and applies hashing to each segment individually. This segmentation allows the system to detect changes in specific portions without requiring complete document matching, thereby improving detection accuracy while maintaining computational efficiency through localized processing.
Solution Approach 2:
The patent employs multiple hashing algorithms with different parameters (e.g., MD5, SHA-1, SHA-256) and varies the segmentation parameters such as segment size and overlap. By changing these parameters dynamically, the system adapts to different document types and change patterns, enhancing detection reliability without sacrificing the simplicity of the underlying hashing mechanism.
2Measurement precision
If whole documents are classified and compared, then complete document similarity can be determined, but identifying and comparing specific portions of user data within documents remains challenging
Solution Approach 1:
The patent automatically segments documents into meaningful portions based on delimiters, data types, or predefined rules. This segmentation enables the system to compare specific portions independently while maintaining the context of the whole document, thus achieving precise portion-level similarity measurement without manual intervention or complex analysis.
Solution Approach 2:
The patent creates a universal hashing framework that can operate at multiple levels: individual portions, combinations of portions, or entire documents. This multi-functional approach allows the same system to handle both granular portion comparison and holistic document comparison, reducing complexity by avoiding separate systems for different comparison scopes.
3Reliability
If statistical functions measuring randomness are applied to subsets of text, then accurate matching and change detection can be achieved, but the computational complexity increases compared to simple hashing
Solution Approach 1:
The patent applies statistical randomness measurements selectively to specific portions of text that are suspected of containing sensitive or change-prone data. Rather than analyzing entire documents uniformly, the system focuses computational resources on critical segments, thereby achieving high detection accuracy while minimizing overall computational complexity through localized analysis.
Solution Approach 2:
The patent implements a two-stage approach where simple hashing is first applied to all document portions for quick comparison, and statistical randomness analysis is applied only to portions that show potential mismatches or contain sensitive data. This partial application of complex analysis reduces overall computational burden while maintaining high detection accuracy for critical changes.
Data Source
AI summary
A method and computing device for statistical data fingerprinting and tracing data similarity of documents. The method comprises applying a statistical function to a subset of text in a first document thereby generating a first fingerprint; applying the statistical function to a subset of text in a second document thereby generating a second fingerprint; comparing the first fingerprint to the second fingerprint; and determining that the subset of text in the first document matches the subset of text in the second document based on the first fingerprint threshold matching the second fingerprint, wherein the statistical function is a measure of randomness of a count of each character in a subset of text against an expected distribution of said characters.