CTPH File Clustering Using Substring Indices for Malware Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing context triggered piecewise hashing (CTPH) clustering techniques for malware detection are computationally expensive and non-deterministic due to the arbitrary choice of similarity metric thresholds, leading to unpredictable results and exponential time complexity.
Innovation Solution
A method that splits CTPH strings into portions and determines indices based on contiguous bits, comparing these indices to existing clusters to associate files, eliminating the need for Levenshtein distance calculations and ensuring deterministic clustering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Levenshtein distance calculation is used for CTPH string comparison, then similarity analysis can be performed, but computational complexity becomes exponential and processing time increases significantly
Solution Approach 1:
The patent divides the CTPH string into multiple substrings of equal length and compares only corresponding substrings between two CTPH strings. This segmentation approach reduces the comparison from O(n²) Levenshtein distance calculation to O(n) substring comparison, where n is the string length, thereby dramatically improving processing speed while maintaining sufficient similarity detection accuracy.
Solution Approach 2:
The patent extracts and compares only the essential portions (substrings) of CTPH strings that are most indicative of similarity, rather than performing comprehensive Levenshtein distance calculation on the entire string. This extraction of key comparison elements reduces computational overhead while preserving the ability to detect meaningful similarities.
2Adaptability or versatility
If arbitrary clustering percentage threshold is used for CTPH string association, then clustering can be performed, but results become unpredictable and non-deterministic
Solution Approach 1:
The patent changes the clustering parameter from an arbitrary percentage threshold to a fixed substring match criterion. Specifically, two CTPH strings are associated if they share at least one matching substring of a predetermined length. This parameter change transforms the clustering process from non-deterministic (dependent on arbitrary threshold choices) to deterministic (based on fixed structural criteria), ensuring consistent and reproducible results.
3Productivity
If traditional block hash techniques are used for malware detection, then detection can be performed, but similarities between malware variants are missed
Solution Approach 1:
The patent applies segmentation to malware detection by dividing CTPH strings into substrings and comparing corresponding substrings between malware samples. This approach detects similarities that traditional block hash techniques miss, as it identifies partial matches and structural similarities even when overall hash values differ significantly, thereby improving detection accuracy for malware variants.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Generally discussed herein are devices, systems, and methods for clustering based on context triggered piecewise hashing (CTPH). A method can include determining a first index of a first CTPH string of the file. The first index can include contiguous bits of the CTPH string. The first index can be smaller than the CTPH string, such as to be a proper subset of the CTPH string. The method can include determining the first index matches a second index of a cluster of files and in response to determining the first index matches the second index of the cluster, associating the file with the cluster. The method can include determining that the file includes malware based on the cluster.