File Similarity Detection Using Two-Phase Vector Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file similarity detection methods suffer from inefficiencies as the number of files to be compared increases, leading to linear complexity and decreased performance, making it impractical to compare a file to large sets effectively.
Innovation Solution
A vector-based approach is employed to break the similarity determination into two phases: detection and estimation, utilizing a vector space to compare a target file with a set of files, allowing for constant complexity (O(1)) regardless of the number of files, with the detection phase identifying similarities quickly and the estimation phase determining the most similar file asynchronously.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional file comparison methods are used to compare a file to a large set of existing files, then similarity detection accuracy is maintained, but the computational complexity increases linearly and performance decreases
Solution Approach 1:
The patent segments the file comparison task into two distinct phases: detection phase and estimation phase. The detection phase uses a simplified approach to quickly identify potential similarities, while the estimation phase provides accurate similarity measurements only when needed. This segmentation resolves the contradiction by maintaining accuracy where necessary while improving overall productivity through the efficient detection phase.
Solution Approach 2:
The patent introduces an intermediary mechanism (the detection phase) that acts as a filter between the initial file and the large set of existing files. This intermediary quickly identifies candidate files that warrant further detailed comparison, thereby maintaining detection accuracy for potential matches while significantly improving productivity by avoiding exhaustive comparisons with all files.
2Reliability
If exhaustive file comparison is performed to ensure accurate similarity detection, then detection reliability is improved, but the time required for comparison increases
Solution Approach 1:
The patent applies preliminary action by performing a quick detection phase before the detailed estimation phase. This preliminary detection identifies files that are likely to be similar based on coarse-grained features, allowing the system to focus detailed analysis only on promising candidates. This approach maintains detection reliability for actual matches while significantly reducing the average comparison time across large file sets.
Solution Approach 2:
The patent implements a dynamic two-phase approach where the level of analysis adapts based on the detection results. Files that pass the detection threshold undergo detailed estimation, while others are quickly discarded. This dynamic strategy ensures reliability for files that require it while minimizing time loss for files that don't warrant detailed analysis.
3Reliability
If a large number of files are compared to comprehensively identify similar files, then detection completeness is improved, but device complexity and computational resources increase
Solution Approach 1:
The patent segments the comprehensive comparison task into detection and estimation phases, each with different complexity requirements. The detection phase uses simpler, more scalable methods to identify candidate files, while the estimation phase provides comprehensive analysis only for those candidates. This segmentation maintains detection completeness for similar files while reducing overall system complexity and computational resource requirements.
4Measurement precision
If detailed similarity estimation is performed for all compared files, then similarity measurement accuracy is improved, but the number of operations required increases
Solution Approach 1:
The patent applies partial action by performing detailed similarity estimation only for files that pass the detection phase, rather than for all files in the set. This selective approach maintains measurement accuracy for files that actually need it while dramatically improving operations efficiency by avoiding unnecessary detailed comparisons with files that are clearly dissimilar.
Data Source
AI summary
A computer-implemented method of detecting similarity between a first file and a plurality of second files, the method includes generating a first vector corresponding to the first file and a plurality of second vectors each corresponding to one of the plurality of second files; determining that the first file is similar to at least one of the plurality of second files based on a comparison of the first vector to the plurality of second vectors; and responsive to determining that the first file is similar to the at least one of the plurality of second files, performing a remediation operation on the first file.