Malware Detection Model Training Using Synthetic Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Building a machine learning model for production use while maintaining data privacy is challenging due to the inability to use production data for training, as it cannot be stored or maintained for privacy reasons, leading to poor model training with lab data that does not match production data.
Innovation Solution
Developing a training dataset using lab or publicly available file samples that match production data by obtaining file identifiers and selecting samples from public repositories based on their similarities, utilizing dimension reduction techniques like tSNE and autoencoders to weight samples closely related to production data features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If production data is used for training machine learning models, then model training quality improves, but data privacy is compromised because production data cannot be stored or maintained
Solution Approach 1:
The patent creates synthetic training data by copying the statistical properties and feature distributions of production data through unsupervised anomaly detection models. These models learn from production data without storing it, then generate synthetic samples that preserve the underlying data characteristics while eliminating privacy risks associated with using actual production data for training.
Solution Approach 2:
The patent introduces an intermediary synthetic data generation process between production data and machine learning training. Instead of directly using production data or relying on imperfect lab data, the system creates synthetic data that acts as a mediator, capturing the essential patterns of production data while removing sensitive information through the anomaly detection framework.
2Object-affected harmful factors
If lab data is used for training machine learning models, then data privacy is maintained, but model training quality deteriorates because lab data does not match production data
Solution Approach 1:
The patent transforms the parameters and feature distributions of synthetic data to match production data characteristics. By adjusting statistical parameters, feature correlations, and data distributions through the anomaly detection framework, the system generates synthetic data that closely resembles production data while maintaining privacy protection, thereby improving model training quality without sacrificing privacy.
3Quantity of substance
If production data is stored for model training, then training dataset availability improves, but data storage requirements increase and privacy control is lost
Solution Approach 1:
The patent creates synthetic copies of production data that capture its essential characteristics without requiring storage of the actual production data. The unsupervised anomaly detection models learn from production data in memory and generate synthetic samples, providing abundant training data availability while eliminating the need for persistent storage and associated privacy management complexity.
Data Source
AI summary
Systems and methods include obtaining file identifiers associated with files in production data; obtaining lab data from one or more public repositories of malware samples based on the file identifiers for the production data; and utilizing the lab data for training a machine learning process for classifying malware in the production data. The obtaining file identifiers can be based on monitoring of users associated with the files, and only the file identifiers are maintained based on the monitoring. The lab data can include samples from the one or more public repositories matching the corresponding file identifiers for the production data. The lab data can include samples from the one or more public repositories that have features closely related to features of the production data.