Inline ML Model Validation Using Realistic Outlier Injection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for validating machine learning models, particularly in security contexts, are inadequate due to the rarity of security events, making continuous validation difficult, and existing approaches like emulation or synthetic data generation fail to accurately reflect real-world scenarios.
Innovation Solution
Incorporate a validation step into the machine learning model pipeline that modifies real-world data to include additional data points simulating security events while maintaining real-world characteristics, using similarity metrics and data modification techniques tailored to categorical and numerical data types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If emulation or synthetic data generation is used for validation, then validation can be performed more frequently, but the accuracy and real-world applicability of validation results deteriorate
Solution Approach 1:
The patent creates synthetic data that copies the statistical properties and patterns of real security events by modifying existing real-world data. Instead of generating completely artificial data, the system copies the essential characteristics (anomaly patterns, event distributions, feature relationships) from real security incidents and embeds them in synthetic datasets, enabling frequent validation while preserving real-world applicability
Solution Approach 2:
The system modifies parameters of existing real-world data to create synthetic security events. By changing specific parameters (adding anomalous patterns, modifying feature values, adjusting event frequencies) while maintaining the overall data structure and relationships, the patent enables controlled generation of validation scenarios that reflect real security conditions without requiring actual security incidents
2Measurement precision
If real-world data is used for validation, then validation accuracy is maintained, but validation frequency is limited due to rarity of security events
Solution Approach 1:
The patent performs preliminary modification of real-world data to create synthetic datasets that contain embedded security events. By preparing these synthetic validation datasets in advance through systematic data modification and augmentation, the system enables frequent validation executions without waiting for rare real security events to occur, thus increasing validation frequency while maintaining accuracy
Solution Approach 2:
The system introduces synthetic data as an intermediary between real-world data and validation processes. This intermediary layer allows the validation model to be tested frequently with realistic scenarios without requiring actual security events, bridging the gap between the rarity of real events and the need for continuous validation
Data Source
AI summary
Methods, apparatuses, and computer readable media are disclosed. An application server may receive a dataset that includes records associated with user device interactions with a computer system. The application server may modify one or more records according to a data modification metric. The modifying may result in a modified dataset that satisfies a similarity metric defining a permissible deviation between the received dataset and the modified dataset according to a deviation threshold. The data modification metric may satisfy the similarity metric and may define a deviation in the modified dataset that results in an expected classification by the machine learning predictive model to classify the deviation in the modified dataset as an outlier event. The application server may process the modified dataset with the machine learning predictive model to produce a result. The application server may compare the expected classification to the classification to validate the model.