Unknown-unknown Detection via Time Series Confidence Intervals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning systems face challenges in detecting unknown or unaccounted data, such as new malware variants, as they may not have been trained on such data, leading to uncertainty in identification.
Innovation Solution
A data processing apparatus and method that utilizes future time series forecasters with confidence intervals to identify unknown-unknowns by determining if new measurements fall outside the generated confidence intervals, labeling them as such, and incorporating these measurements into historical data to improve forecasting accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning systems use traditional classification methods to detect known malware variants, then detection accuracy for known threats is improved, but the system cannot identify new malware variants that were not included in training data
Solution Approach 1:
The system dynamically adapts by continuously monitoring system behavior through time series forecasters and adjusting to new patterns. The confidence interval mechanism allows the system to dynamically identify when new malware variants are detected by recognizing measurements outside expected ranges, enabling real-time adaptation without retraining on new threat data
Solution Approach 2:
The system performs preliminary analysis by establishing confidence intervals from training data before encountering new threats. This preliminary configuration of detection boundaries enables the system to quickly identify novel malware variants by comparing new measurements against pre-established normal behavior ranges, allowing rapid response to new threats
2Adaptability or versatility
If the system expands training data to cover more malware variants, then detection coverage is improved, but the system complexity and data processing requirements increase
Solution Approach 1:
The time series forecaster serves multiple functions: it predicts future system states, generates confidence intervals for anomaly detection, and adapts to new malware variants without requiring separate detection mechanisms. This multi-functional approach expands detection coverage while avoiding the complexity of multiple specialized systems
Solution Approach 2:
The system changes parameters by adjusting confidence intervals based on forecast uncertainty rather than expanding training datasets. This parameter-based adaptation allows the system to handle diverse malware variants by modifying detection thresholds dynamically, maintaining simplicity while improving coverage
Data Source
AI summary
A data processing apparatus is provided that includes storage circuitry that stores a plurality of future time series forecasters of an aspect of a system and, for each of the future time series forecasters, a representation of a confidence interval associated with that future time series forecaster. Unknown-unknown detection circuitry determines whether a new measurement falls outside confidence intervals generated from the representation of the confidence interval associated with each future time series forecaster of the aspect of the system, and in response to the new measurement falling outside the confidence intervals, labels the new measurement as an unknown-unknown.