Dynamic Threshold Selection for Malicious Activity Detection Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing systems face challenges in detecting malicious activity due to changing patterns and functionalities, leading to degraded model performance over time, and require manual threshold adjustments to manage false positives and negatives.
Innovation Solution
A system that automatically trains and evaluates candidate machine learning models against a production model, dynamically sets alert and reset thresholds for each urgency level, and promotes the best performing model to the production system to accommodate changing malicious behaviors and system functionalities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual threshold adjustments are used to manage false positives and negatives, then alert quality can be maintained, but system complexity and time consumption increase due to iterative adjustments
Solution Approach 1:
The system automatically selects thresholds and promotes models without human intervention. The model evaluation module autonomously compares candidate models against the production model using test data, and the threshold selection module automatically determines optimal thresholds based on performance metrics, eliminating the need for manual threshold adjustments while maintaining alert quality
Solution Approach 2:
The system implements a feedback loop where model performance is continuously monitored using test data that reflects current malicious activity patterns. Based on this feedback, the system automatically selects thresholds that optimize the balance between false positives and false negatives, and promotes improved models to production, creating a self-correcting system that maintains reliability without manual intervention
2Measurement precision
If new models are trained frequently to accommodate changing malicious behaviors, then detection accuracy improves, but system complexity and training time increase
Solution Approach 1:
The system trains multiple candidate models in advance before they are needed for production. These candidate models are trained on historical data and evaluated against test data that reflects current malicious activity patterns. When a candidate model outperforms the current production model, it is automatically promoted, ensuring the system is always prepared with updated detection capabilities without requiring complex on-demand training processes
Solution Approach 2:
The system creates multiple candidate model copies that can be trained and evaluated in parallel against the same test data. This allows the system to explore multiple model variations simultaneously and select the best performing one for promotion, improving detection accuracy through diverse model comparisons while distributing the training complexity across multiple independent copies rather than requiring complex sequential training
3Reliability
If manual model evaluation and promotion processes are used, then model performance can be monitored, but productivity and response time decrease due to manual observations
Solution Approach 1:
The model evaluation module automatically evaluates candidate models against test data without human intervention. The system autonomously computes performance metrics, compares candidate models to the production model, and identifies the best performing candidate. The promotion module then automatically promotes the selected model to production, eliminating manual observation and deployment steps while maintaining reliable performance monitoring through automated metric tracking
Solution Approach 2:
The system implements continuous automated model evaluation and promotion processes that operate without interruption. Candidate models are continuously trained and evaluated against updated test data, and improvements are continuously promoted to production when detected. This continuous automated process maintains reliable performance monitoring while significantly increasing productivity compared to discrete manual evaluation and promotion cycles
Data Source
AI summary
A set of candidate malicious activity identification models are trained and evaluated against a production malicious activity identification model to identify a best performing model. If the best performing model is one of the candidate models, then an alert threshold is dynamically set for the best performing model, for each of a plurality of different urgency levels. A reset threshold, for each urgency level, is also dynamically set for the best performing model.