Anomaly Detection Model Evaluation via Aggregated Time-Series Signals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection models in information processing systems face challenges in reliably evaluating their performance, particularly when dealing with unsupervised learning models that lack target output labels and require noise reduction in aggregate time-series signals to enhance predictability.
Innovation Solution
A method is introduced that involves generating a reduced set of transactions based on anomaly scores, aggregating them to create an aggregated time-series signal, training a forecast algorithm, calculating performance metrics, and initiating automated actions to improve predictability and model selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If anomaly detection models are applied to detect anomalies in discrete transactions, then anomaly detection capability is improved, but reliable evaluation of model performance becomes difficult
Solution Approach 1:
The patent introduces an aggregated time-series signal as an intermediary between the anomaly detection model and the evaluation process. Instead of evaluating individual transaction predictions directly, the model's anomaly scores are applied to aggregate the time-series signal, and forecast accuracy on this aggregated signal serves as the evaluation metric. This intermediary approach enables reliable performance measurement for unsupervised anomaly detection models.
Solution Approach 2:
The patent transitions from evaluating anomaly detection at the individual transaction level to evaluating it at the aggregated time-series level. By changing the dimension of evaluation from discrete transaction predictions to continuous aggregated signal forecast accuracy, the system enables meaningful performance assessment of unsupervised anomaly detection models.
2Quantity of substance
If all discrete transactions are included in the aggregated time-series signal, then data completeness is improved, but predictability and noise reduction deteriorate
Solution Approach 1:
The patent extracts anomalous transactions from the complete dataset by applying the anomaly detection model and removing transactions with high anomaly scores before aggregation. This extraction process reduces noise in the aggregated time-series signal while maintaining the majority of normal transaction data, thereby improving predictability without completely sacrificing data completeness.
Solution Approach 2:
The patent applies partial action by removing only the most anomalous transactions (those exceeding a certain score threshold) rather than filtering all data or removing a fixed proportion. This selective removal approach maintains data completeness for normal transactions while eliminating sufficient noise to improve predictability.
Data Source
AI summary
Techniques are provided for evaluating one or more anomaly detection models using aggregated time-series signals. One method comprises obtaining discrete transactions; determining feature values for the discrete transactions; applying the feature values to at least one anomaly detection model that generates an anomaly score for each discrete transaction; generating a reduced set of the discrete transactions using the anomaly score for each of the plurality of discrete transactions; aggregating the discrete transactions of the reduced set to create an aggregated time-series signal; training a forecast algorithm using a first portion of the aggregated time-series signal; generating a prediction of a second portion of the aggregated time-series signal using the trained forecast algorithm; calculating a performance metric of the forecast algorithm based on a difference between: the second portion of the aggregated time-series signal and the prediction of the second portion; and initiating an automated action using the performance metric.