CTPH Index Clustering for Deterministic Malware Similarity Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Context Triggered Piecewise Hashing (CTPH) clustering techniques are computationally expensive and non-deterministic, relying on arbitrary Levenshtein distance thresholds, which leads to unpredictable results and excessive computational resources in similarity analysis for malware detection.
Innovation Solution
The proposed solution involves comparing indices of CTPH strings instead of calculating Levenshtein distances, associating files based on matching indices, and using multiple indices (most significant, second most significant, etc.) to determine cluster membership, thereby reducing computational complexity and introducing determinism in clustering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Levenshtein distance calculation is used to determine similarity between CTPH strings, then similarity analysis can be performed, but computation resources and computation time become prohibitively expensive
Solution Approach 1:
The CTPH string is divided into multiple substrings of equal length, and separate hashes are computed for each substring. This segmentation allows the system to compare only specific portions of files rather than performing full Levenshtein distance calculations, significantly reducing computation time while maintaining similarity detection accuracy.
Solution Approach 2:
The patent extracts specific substrings from CTPH strings at predetermined offsets and uses only these extracted portions for comparison. By taking out and comparing only the necessary substrings rather than the entire strings, the system achieves fast similarity analysis without requiring expensive full-string Levenshtein distance calculations.
2Ease of operation
If Levenshtein distance threshold is used to determine clustering, then file similarity can be determined, but results become arbitrary and unpredictable
Solution Approach 1:
The patent changes the parameter used for clustering from continuous Levenshtein distance to discrete hash value equality. By comparing whether hash values of substrings are exactly equal rather than calculating continuous distances and applying arbitrary thresholds, the system achieves deterministic clustering results that are reproducible and predictable.
3Measurement precision
If multiple substrings are compared for clustering membership, then clustering accuracy improves, but computational complexity increases
Solution Approach 1:
The patent computes hashes of multiple substrings and stores them in advance for each CTPH string. During clustering, these pre-computed hashes are simply compared for equality rather than recalculating distances. This preliminary action of hashing substrings beforehand reduces the complexity of the actual clustering operation to simple equality checks.
Data Source
AI summary
Generally discussed herein are devices, systems, and methods for clustering based on context triggered piecewise hashing (CTPH). A method can include determining a first index of a first CTPH string of the file. The first index can include contiguous bits of the CTPH string. The first index can be smaller than the CTPH string, such as to be a proper subset of the CTPH string. The method can include determining the first index matches a second index of a cluster of files and in response to determining the first index matches the second index of the cluster, associating the file with the cluster. The method can include determining that the file includes malware based on the cluster.