Malware Training Set Clustering for Duplicate Sample Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for training machine learning models to detect malicious code are inefficient due to the presence of duplicated and mutated malware, which are treated as new threats, leading to biased models and increased processing overhead, as they fail to account for the similarity and mutation of malicious code effectively.
Innovation Solution
A computer-implemented method that generates reduced dimensionality vectors from original feature vectors, applies clustering algorithms to identify and refine clusters, and selects exemplars from these clusters to create a training set for a machine learning model, thereby reducing the impact of duplicates and mutations in the training dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional methods process all training samples including duplicates, then the model training is comprehensive, but the processing overhead increases significantly and training efficiency decreases
Solution Approach 1:
The patent extracts and removes duplicate samples from the training set before model training. By identifying and eliminating redundant malicious code samples that are substantially similar to existing ones, the system reduces the training dataset size while maintaining representativeness, thereby improving training efficiency and reducing processing overhead without sacrificing model comprehensiveness
Solution Approach 2:
The patent changes the parameter of sample selection by introducing similarity thresholds and diversity metrics. Instead of processing all samples uniformly, the system applies parameter-based filtering to identify duplicates exceeding a similarity threshold and selectively removes them, transforming the training process from exhaustive to optimized based on sample characteristics
2Measurement precision
If the training dataset includes many duplicates of specific malware, then the model learns that malware type well, but the model becomes biased toward that type and performs poorly on other malware types
Solution Approach 1:
The patent applies local quality by treating different malware types differently in the training process. Instead of uniform sampling, the system identifies clusters of similar malware and ensures balanced representation across different malware families by controlling the number of samples selected from each cluster, thereby improving both specific detection accuracy and overall model versatility
Solution Approach 2:
The patent changes the sampling parameters by introducing diversity constraints and cluster-based selection criteria. The system adjusts sample selection parameters to maintain a balanced distribution across different malware types, preventing any single type from dominating the training data and causing model bias
3Reliability
If conventional methods treat mutated malware as entirely new threats, then the model processes all variations, but the training becomes inefficient and computationally expensive
Solution Approach 1:
The patent extracts and identifies mutated versions of malware by comparing them against existing training samples using similarity metrics. Mutated samples that exceed a similarity threshold are identified as duplicates and removed from the training set, allowing the model to learn from representative samples while efficiently handling variations through the similarity-based filtering mechanism
Data Source
AI summary
A method, a system, and a computer program product for performing analysis of data to detect presence of malicious code are disclosed. Reduced dimensionality vectors are generated from a plurality of original dimensionality vectors representing features in a plurality of samples. The reduced dimensionality vectors have a lower dimensionality than an original dimensionality of the plurality of original dimensionality vectors. A first plurality of clusters is determined by applying a first clustering algorithm to the reduced dimensionality vectors. A second plurality of clusters is determined by applying a second clustering algorithm to one or more clusters in the first plurality of clusters using the original dimensionality. An exemplar for a cluster in the second plurality of clusters is added to a training set, which is used to train a machine learning model for identifying a file containing malicious code.