Malware Training Set Clustering for Deduplication and Bias Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for training machine learning models to detect malicious code are inefficient due to the presence of duplicated and mutated malware, which are treated as new threats, leading to biased models and increased processing overhead, as they fail to account for the similarity and mutation of malicious code effectively.
Innovation Solution
A computer-implemented method that generates reduced dimensionality vectors from original feature vectors, applies clustering algorithms to identify and refine clusters, and selects exemplars from these clusters to create a training set for a machine learning model, thereby reducing the impact of duplicates and mutations in the training dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional methods process all samples in the training dataset, then the model training is comprehensive, but the processing time and computational resources increase significantly due to duplicated and mutated malware samples
Solution Approach 1:
The patent segments the training dataset into clusters of duplicated and mutated malware samples using clustering algorithms. By grouping similar samples together, the system processes only representative exemplars from each cluster rather than every individual sample, significantly reducing training time while maintaining model comprehensiveness through the representative selection strategy.
Solution Approach 2:
The patent creates simplified representations (exemplars) of malware sample clusters that capture the essential characteristics of each group. These exemplars serve as copies that represent entire clusters of duplicated and mutated samples, allowing the model to learn from condensed versions that preserve the core patterns without the redundancy of processing every original sample.
2Measurement precision
If the training dataset includes many duplicated malware samples, then the model learns about specific malware types, but the model becomes biased toward those types and loses adaptability to other malware
Solution Approach 1:
The patent extracts and removes the harmful effect of duplicated samples by identifying and clustering near-duplicate malware samples, then selecting only one exemplar from each cluster for training. This extraction process eliminates the biasing effect of duplicates while preserving the essential characteristics of each malware type, allowing the model to learn diverse malware patterns without being overrepresented by any single type.
Solution Approach 2:
The patent changes the parameter of sample representation by transforming individual malware samples into cluster-based exemplars. This parameter change from individual samples to representative cluster members adjusts the training data distribution to be more balanced across different malware types, improving model adaptability while maintaining detection precision through the preservation of key malware characteristics in the exemplars.
3Reliability
If conventional methods treat mutated malware as new threats, then the model processes all variations, but the processing overhead increases and efficiency decreases
Solution Approach 1:
The patent merges mutated malware samples that are identified as near-duplicates into unified clusters, treating them as variations of the same underlying threat rather than entirely separate threats. By combining these related samples into single clusters and selecting representative exemplars, the system maintains comprehensive threat detection coverage while significantly improving processing efficiency through reduced redundancy.
Data Source
AI summary
A method, a system, and a computer program product for performing analysis of data to detect presence of malicious code are disclosed. Reduced dimensionality vectors are generated from a plurality of original dimensionality vectors representing features in a plurality of samples. The reduced dimensionality vectors have a lower dimensionality than an original dimensionality of the plurality of original dimensionality vectors. A first plurality of clusters is determined by applying a first clustering algorithm to the reduced dimensionality vectors. A second plurality of clusters is determined by applying a second clustering algorithm to one or more clusters in the first plurality of clusters using the original dimensionality. An exemplar for a cluster in the second plurality of clusters is added to a training set, which is used to train a machine learning model for identifying a file containing malicious code.