Malware Detection Training Data Pruning for Generalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing malware detection models face challenges with overwhelming data volumes, computational strain, and overfitting due to data redundancy, which affects their scalability and generalization.
Innovation Solution
Implement a sequential sampling strategy that includes threshold sampling, top-n most recent and least confident data sampling, and Monte Carlo sampling with power transformation to prune training data, focusing on rare and recent samples while removing redundant data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data handling techniques are used to process malware detection data, then the model can learn from comprehensive datasets, but computational resources are strained and processing time increases
Solution Approach 1:
The patent extracts and removes redundant data from the malware detection dataset through a multi-stage pruning process. First, duplicate files are identified and removed using hash-based comparison. Then, statistically insignificant samples are filtered out using confidence scoring and variance analysis. This extraction of essential information while discarding redundancy reduces computational resource consumption while preserving detection accuracy.
Solution Approach 2:
The patent changes parameters of the training data by applying transformations such as downsampling high-frequency redundant features, normalizing confidence score distributions, and adjusting sample weights based on rarity metrics. These parameter changes optimize the data representation to reduce computational strain while maintaining or improving detection reliability.
2Adaptability or versatility
If comprehensive datasets are used for training malware detection models, then model coverage is improved, but overfitting occurs due to data redundancy
Solution Approach 1:
The patent extracts and removes redundant data patterns that cause overfitting through duplicate detection using cryptographic hash comparison and statistical analysis of sample distributions. By eliminating exact duplicates and near-duplicates, the model is forced to learn from diverse, unique patterns rather than memorizing redundant examples, thereby improving generalization while maintaining coverage.
Solution Approach 2:
The patent applies selective pruning that removes only the excessive redundant portion of the dataset while preserving the essential diverse samples needed for comprehensive coverage. Through confidence-based filtering and rarity-weighted sampling, it retains just enough variation to maintain adaptability while discarding the excessive redundancy that causes overfitting.
3Quantity of substance
If all detected malware data is retained for training, then data completeness is maintained, but data redundancy increases and processing efficiency decreases
Solution Approach 1:
The patent extracts and removes redundant data through a systematic pruning process that identifies and eliminates duplicate files using hash comparison, filters out statistically insignificant samples using confidence scoring, and removes near-duplicates through similarity analysis. This extraction maintains the essential quantity of diverse malware patterns while removing redundant copies that hinder training efficiency.
Solution Approach 2:
The patent discards redundant data during the pruning process while maintaining the ability to recover essential information through careful selection criteria. Samples are discarded based on redundancy metrics but the pruning process preserves rare and diverse patterns that are critical for detection, effectively discarding only what is unnecessary while recovering and retaining what is valuable.
4Productivity
If sequential sampling strategy is applied to prune training data, then redundant data is removed and computational load is reduced, but data selection complexity increases
Solution Approach 1:
The patent segments the data pruning process into distinct sequential stages: duplicate detection using hash comparison, confidence-based filtering using statistical scoring, rarity analysis using distribution metrics, and final sampling selection. This segmentation breaks down the complex task of data pruning into manageable, systematic steps that improve training speed while keeping each stage's complexity controlled and understandable.
Data Source
AI summary
Methods and systems for implementing enhanced data pruning strategy for malware detection models are described herein. According to an implementation, a computer device may distribute data associated with detected events into a plurality of storages. The computer device may sequentially perform one or more sampling operations to construct a dataset for malware detection model training. The computer device may first select a subset of the plurality of storages, each having a size equal to or less than a threshold, to be used for model training without pruning. The computer device may then select top-n most recent samples and top-n least confident samples from each of rest storages. Further, the computer device may perform Monte Carlo sampling enhanced with a power transformation on the rest storages to generate additional samples. The compute device may then generate the training dataset for the malware detection model training based on the sequentially sampling results.