Dictionary-Based Deduplication for ML Malware Training Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for detecting malicious code in machine learning models are inefficient in handling duplicated or mutated malware, treating them as new threats rather than recognizing their similarity to existing code, which leads to inefficient training and resource consumption.
Innovation Solution
A method involving vectorization, dimension reduction, and binarization of feature sets from data samples, using a dictionary structure to identify and eliminate duplicate binary representations, thereby creating a reduced training set for machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional methods treat duplicated/mutated malicious code as entirely new malicious code, then detection coverage is maintained, but training efficiency and resource consumption deteriorate
Solution Approach 1:
The patent segments the training process into two distinct phases: (1) a preprocessing phase that extracts features from training samples and identifies duplicates using similarity comparison, and (2) a machine learning training phase that uses only the deduplicated samples. This segmentation allows duplicate detection to occur before training, preventing redundant processing while maintaining comprehensive detection coverage.
Solution Approach 2:
The patent performs preliminary deduplication of training samples before the actual machine learning training process. By extracting features, comparing similarities, and removing duplicates in advance, the system prepares a optimized training set that maintains detection coverage while significantly reducing training time and computational resources.
2Quantity of substance
If all training samples including duplicates are used, then comprehensive training data is available, but storage requirements and processing time increase
Solution Approach 1:
The patent changes the parameter of training data representation by transforming raw samples into feature vectors, then comparing these vectors to identify duplicates. This parameter transformation enables efficient duplicate detection without requiring full processing of redundant samples, thereby reducing processing time while maintaining adequate training data volume through selective retention of unique samples.
3Productivity
If duplicate malicious code samples are removed from training set, then training efficiency improves, but detection of mutated malware may be affected
Solution Approach 1:
The patent replaces the mechanical approach of training on all raw samples with a systematic feature-based comparison mechanism. By extracting meaningful features from samples and using similarity comparison algorithms, the system can identify true duplicates while preserving samples with meaningful variations, thus maintaining mutated malware detection capability while improving training efficiency.
Data Source
AI summary
Presence of malicious code can be identified in one or more data samples. A feature set extracted from a sample is vectorized to generate a sparse vector. A reduced dimension vector representing the sparse vector can be generated. A binary representation vector of reduced dimension vector can be created by converting each value of a plurality of values in the reduced dimension vector to a binary representation. The binary representation vector can be added as a new element in a dictionary structure if the binary representation is not equal to an existing element in the dictionary structure. A training set for use in training a machine learning model can be created to include one vector whose binary representation corresponds to each of a plurality of elements in the dictionary structure.