MinHash Similarity Hashing for Binary File Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Malicious binary file detection faces scalability challenges and loses metadata and functionality information when using traditional hashing schemes, which are not effective in capturing similarity in binary files.
Innovation Solution
The system employs MinHash locality sensitive hashing to generate similarity hashes from feature sets of binary files, allowing for accurate and scalable detection by clustering and labeling binary files based on approximate nearest neighbor searches, and updates clusters with ground truth labels to identify indicators of compromise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If traditional hashing schemes are used to compare binary files, then the hashing process is simple and fast, but the similarity information and functionality information of binary files are lost
Solution Approach 1:
The binary file is segmented into multiple feature sets (e.g., metadata features, code block features, control flow graph features). Each feature set is independently hashed using MinHash, and the results are combined into a composite hash vector. This segmentation allows preservation of different aspects of binary file information while maintaining computational efficiency.
Solution Approach 2:
MinHash locality-sensitive hashing is introduced as an intermediary between traditional hashing and similarity comparison. Instead of directly comparing binary files or using complex similarity metrics, MinHash transforms feature sets into hash vectors that preserve similarity information, enabling efficient approximate nearest neighbor search while retaining functional similarity data.
2Measurement precision
If traditional hashing schemes are used for binary file detection, then the processing speed is fast, but the detection accuracy for similar malicious files is poor
Solution Approach 1:
The hashing approach is changed from traditional cryptographic hashing to MinHash locality-sensitive hashing. This parameter change in the hashing function enables similar binary files (including obfuscated malware variants) to produce similar hash vectors, significantly improving detection accuracy for similar malicious files while maintaining processing efficiency through the properties of LSH.
Solution Approach 2:
The system transitions from single-hash comparison to multi-dimensional hash vector comparison. By generating multiple MinHash values for each feature set and combining them into a composite hash vector, the system creates a richer representation space that captures nuanced similarities between binary files, improving detection precision without proportionally increasing computational burden.
3Loss of information
If MinHash locality sensitive hashing is applied to feature sets of binary files, then similarity information is captured accurately, but the computational complexity and processing time increase
Solution Approach 1:
The binary file analysis is segmented into extracting multiple feature sets (metadata, code blocks, control flow graphs), with MinHash applied independently to each feature set. This segmentation allows parallel processing of different feature extractions and hashing operations, capturing comprehensive functionality information while managing computational complexity through modular, independent processing stages.
Data Source
AI summary
Locality sensitive hashing of feature sets generated from disassembly binary files results in hashes that capture similar and dissimilar functionality across binary files. Comparing hashes of binary files allows for malicious detection by identifying binary files with similar hashes to known malicious binary files. Scalable storage and clustering of hashes using approximate nearest neighbor search in a vector database allows for classification of large stores of binary files according to cluster labels. Storage of verdicts from the clustering and other metadata in a non-relational database further allows for scalable analysis of strata of binary files according to criteria on complex binary file metadata.