MinHash LSH Clustering for Malware Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current malware detection methods are inefficient in handling polymorphic malware variants and zero-day attacks, as they rely on signature-based approaches that are slow to update and struggle with scalability, leading to prolonged vulnerability in computer systems.
Innovation Solution
The implementation of a probabilistic algorithm using MinHash Locality Sensitive Hashing (LSH) for clustering malware based on static attributes, allowing for sublinear complexity and efficient detection of similar objects, combined with machine learning for real-time classification and dynamic analysis of packed files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If classical clustering algorithms (DBSCAN, Hierarchical) are used for malware detection, then clustering accuracy is improved, but computational complexity increases to O(n²) or O(n³)
Solution Approach 1:
The patent segments the clustering process into two distinct phases: rough clustering using MinHash LSH with sublinear complexity to group similar malware samples, and then exact clustering on the reduced subsets. This segmentation allows the system to achieve accurate clustering results without computing pairwise distances for all n samples, thereby resolving the contradiction between clustering accuracy and computational complexity.
Solution Approach 2:
The patent introduces MinHash LSH as an intermediary mechanism between the raw malware samples and the final clustering results. This intermediary performs approximate similarity computation with sublinear complexity, filtering out dissimilar samples before they enter the expensive exact clustering phase, thus enabling accurate clustering at reduced computational cost.
2Measurement precision
If signature-based malware detection is used, then detection accuracy for known malware is improved, but response time increases due to slow update cycles
Solution Approach 1:
The patent performs preliminary action by pre-computing MinHash signatures and organizing malware samples into clusters before actual detection is needed. When new malware samples arrive, the system can quickly perform LSH-based similarity search without waiting for signature updates, enabling real-time detection of both known and novel malware variants while maintaining high accuracy.
Solution Approach 2:
The patent creates compact copies of malware characteristics through MinHash signatures, which are much smaller and faster to process than full malware signatures. These signature copies enable rapid comparison and clustering operations, reducing the time penalty associated with traditional signature-based detection while preserving detection accuracy.
3Productivity
If static analysis alone is used for malware detection, then processing speed is improved, but detection accuracy decreases for packed malware
Solution Approach 1:
The patent implements a dynamic, multi-stage analysis system that adapts its depth based on initial findings. The system begins with fast static analysis using MinHash LSH clustering, then dynamically transitions to more intensive analysis (including dynamic execution) only for samples that exhibit packing characteristics or remain unclassified, thus optimizing the balance between processing speed and detection accuracy.
Solution Approach 2:
The patent applies partial action by performing comprehensive static analysis only on a subset of malware samples identified through LSH clustering, while applying more resource-intensive dynamic analysis selectively. This partial application of exhaustive analysis methods maintains high overall processing throughput while achieving accurate detection of packed malware in the targeted subset.
Data Source
AI summary
A system and method for detecting malware using hierarchical clustering analysis. Unknown files classified by clustering and in view of known malicious and known safe files. A search is made for similar files using the probabilistic MinHash LSH algorithm applying a Jaccard measure. Machine learning models and detection rules are used to enhance classification accuracy.