CTPH File Clustering Using Hash Indices Instead of Edit Distance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Context Triggered Piecewise Hashing (CTPH) clustering techniques are computationally expensive and non-deterministic, relying on arbitrary Levenshtein distance thresholds for similarity analysis, which leads to unpredictable results and excessive resource usage in malware detection and similarity analysis.
Innovation Solution
The proposed solution involves comparing indices of CTPH strings instead of calculating Levenshtein distances, allowing for deterministic clustering by associating files based on matching indices, thereby reducing computational complexity and resource usage while improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Levenshtein distance calculation is used for CTPH string comparison, then similarity analysis can be performed, but computation resources and computation time become prohibitively expensive
Solution Approach 1:
The patent segments the CTPH string comparison task by extracting and comparing only specific indices (contiguous bits) rather than performing full Levenshtein distance calculation. This segmentation allows the system to maintain similarity analysis capability while dramatically reducing computational complexity and processing time.
2Adaptability or versatility
If Levenshtein distance threshold is used for clustering determination, then clustering can be performed, but the result becomes arbitrary and unpredictable
Solution Approach 1:
The patent extracts specific indices from CTPH strings and uses these extracted elements for deterministic comparison and clustering. By taking out and comparing only the essential index portions, the system achieves both clustering functionality and deterministic, predictable results without relying on arbitrary distance thresholds.
3Measurement precision
If full CTPH string comparison is performed, then accurate similarity detection is achieved, but memory overhead increases
Solution Approach 1:
The patent extracts and stores only the essential indices (contiguous bits) of CTPH strings for comparison purposes. This extraction approach maintains the ability to detect similar files accurately while significantly reducing the memory overhead required to store and process the hash data.
Data Source
AI summary
Generally discussed herein are devices, systems, and methods for clustering based on context triggered piecewise hashing (CTPH). A method can include determining a first index of a first CTPH string of the file. The first index can include contiguous bits of the CTPH string. The first index can be smaller than the CTPH string, such as to be a proper subset of the CTPH string. The method can include determining the first index matches a second index of a cluster of files and in response to determining the first index matches the second index of the cluster, associating the file with the cluster. The method can include determining that the file includes malware based on the cluster.