Semantic Meshing for Digital Data Similarity Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining similarity among digital data files face challenges such as file format impedance, performance limitations, sample window alignment issues, and comprehensibility problems, leading to inefficient detection of similar documents across disparate files.
Innovation Solution
The method employs a multi-character rolling window, uniform hash functions, and a force-directed mesh to normalize and compare digital data streams, identifying semantic hashes that represent document similarity, while optimizing processing resources and presenting results in a comprehensible format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If block-based sampling is used to detect file similarity, then the method can handle large volumes of content, but insertions or deletions in the middle of a file shift which bytes/characters are in subsequent sampled blocks, leading to misalignment
Solution Approach 1:
The patent segments the file into blocks and applies rolling hash functions to each block independently. This segmentation allows the system to handle large files efficiently while maintaining alignment within each block, as the hash computation is localized to specific segments rather than requiring global alignment.
Solution Approach 2:
The patent performs preliminary normalization of the file content before sampling, converting all text to a consistent format (e.g., lowercase, normalized whitespace). This preliminary action ensures that subsequent block-based sampling operates on standardized data, reducing alignment issues caused by formatting variations.
2Ease of manufacture
If simple similarity detection implementations are used, then the system is easy to implement, but performance limitations increase sharply as more and larger documents are added
Solution Approach 1:
The patent replaces direct string comparison mechanisms with cryptographic hash function computations. Instead of comparing actual file contents byte-by-byte, the system computes hash values (e.g., SHA-256) for each block and compares these compact representations. This substitution maintains implementation simplicity while dramatically improving performance for large document volumes.
Solution Approach 2:
The patent changes the parameter being compared from raw file content to hash values of file blocks. By transforming the comparison task from operating on large binary data to operating on fixed-size hash representations, the system achieves better performance scalability while keeping the implementation relatively simple.
3Adaptability or versatility
If different software vendors use different file formats, then each vendor can optimize their own format, but file format impedance makes detecting similarity between files from different vendors significantly more complex
Solution Approach 1:
The patent implements a universal preprocessing layer that can handle multiple file formats (PDF, DOCX, TXT, HTML, etc.) by converting them all to a common normalized text representation. This universal approach allows the same similarity detection algorithm to work across vendor-specific formats without requiring format-specific logic, reducing overall system complexity.
Solution Approach 2:
The patent introduces an intermediary normalization step between file ingestion and similarity detection. This intermediary layer converts diverse vendor-specific formats into a standardized intermediate representation, allowing the core similarity detection algorithm to operate uniformly across all formats without directly handling format-specific complexities.
4Ease of operation
If microarray heatmaps are used to visualize similarity relationships, then many-to-many comparisons can be displayed, but every cell in the array needs to be calculated which is very expensive in terms of processing resources
Solution Approach 1:
The patent computes similarity hashes for all document blocks (excessive action) but uses these pre-computed hashes to quickly determine which document pairs warrant full similarity calculation. This partial computation approach avoids calculating every cell in a microarray heatmap while still enabling comprehensive similarity detection for relevant document pairs.
Solution Approach 2:
The patent performs preliminary hash computation for all document blocks before generating visualizations. These pre-computed hashes serve as filters to identify candidate similar documents, allowing the system to avoid expensive full similarity calculations for document pairs that are clearly dissimilar, thereby reducing processing resource requirements.
5Productivity
If subsample cluster analysis is used, then a subset of matched documents can be analyzed, but it simply renders a list of relationships without clear visualization
Solution Approach 1:
The patent transforms the flat list of similarity relationships into a force-directed graph visualization that adds spatial dimensionality. Documents and their similarity relationships are represented as nodes and edges in a 2D layout, where connected components visually cluster together. This dimensional transformation makes the results more comprehensible by revealing structural patterns that are not apparent in linear lists.
Data Source
AI summary
In at least one embodiment, the present invention provides methods and systems for improved semantic meshing, comprising receiving an input data stream consisting of a plurality of characters; generating a normalized stream having an initial value based on said input data stream; applying a plural character rolling window to a subset of the normalized stream to select at least one stream subset, applying a first uniform hash function to the at least one stream subset to create at least one digest, identifying a cut if the modulus of the digest is zero, such that identifying a cut includes applying a second uniform hash function to the remainder values of the normalized stream, generating at least one shingle, resetting the plural character rolling window with a plurality of zeros and aggregating the at least one shingle into a semantic hash.