Supervised Threat Detection Retraining With Unsupervised Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional computer systems are vulnerable to unpermitted access, and threat detection methods often fail to prevent data breaches until it's too late to remediate the damage.
Innovation Solution
A system combining unsupervised and supervised machine learning models to label unlabeled data, using unsupervised learning to generate initial labels and supervised learning to refine predictions, with a label audit process to update models for improved threat detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional threat detection methods are used, then the system can identify unpermitted access, but the detection occurs too late to prevent data breaches and remediate damage
Solution Approach 1:
The system performs unsupervised learning analysis on network traffic data before threats fully manifest, establishing baseline behavior patterns and identifying potential anomalies in advance. This preliminary action enables the system to detect suspicious activities earlier in the threat lifecycle, allowing preventive measures to be taken before data breaches occur.
Solution Approach 2:
The system implements a feedback loop where detected threats and false positives are continuously fed back into the machine learning models for retraining. This feedback mechanism improves detection accuracy over time and reduces false alarms, enabling more reliable real-time threat prevention while maintaining rapid response capabilities.
2Measurement precision
If supervised learning models are used alone, then the system can classify known threats, but it fails to detect novel or zero-day attacks
Solution Approach 1:
The system merges supervised learning models (for classification of known threats) with unsupervised learning models (for detection of novel patterns). The supervised component provides precise classification of identified threat types, while the unsupervised component continuously monitors for deviations from baseline behavior that may indicate zero-day attacks, achieving both precision and adaptability.
Solution Approach 2:
The system dynamically adjusts the balance between supervised and unsupervised learning based on threat landscape changes. When new attack patterns are detected by the unsupervised model, the system adapts by incorporating these patterns into supervised models, creating a dynamic system that evolves its detection capabilities in response to emerging threats.
3Measurement precision
If more data is collected for analysis, then the system can improve detection accuracy, but the processing time and computational resources increase
Solution Approach 1:
The system extracts and focuses on the most relevant features and indicators from large volumes of network traffic data using unsupervised learning. By identifying and prioritizing key anomaly indicators rather than analyzing all raw data equally, the system maintains high detection accuracy while significantly reducing processing time and computational resource requirements.
Solution Approach 2:
The system applies partial analysis to most data points (using lightweight unsupervised filtering) and excessive/detailed analysis only to suspicious data points that trigger anomaly detection. This selective approach allows comprehensive monitoring with minimal processing overhead, maintaining accuracy while enabling real-time analysis.
Data Source
AI summary
Systems and methods are described for automated threat detection. For example, the system receives various types of unlabeled data and determines, through an unsupervised machine learning model, a label for the data. The labels are provided to a supervised machine learning model during a first training process. When new data is received, the supervised machine learning model is executed during an inference process to cluster the new data in accordance with the labels that were determined by the unsupervised machine learning model. In some examples, a label audit process may be implemented to update the cluster/output of the supervised machine learning model. The updated labels from the label audit process may be provided back to the supervised machine learning model during a second training process. In other words, the system may combine the unsupervised machine learning model with a supervised machine learning model to perform automated threat detection.