Anomaly Detection Model Using Synthetic Data for Outage Gaps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Anomaly detection models are impacted by data outages, leading to inaccurate predictions and missed anomalies during the outage and false positives upon connectivity restoration due to missing data patterns and data surges.
Innovation Solution
A computer-implemented method that builds an anomaly detection model using characterized sensor data, marks sensor states during outages, identifies missing data using predictive modeling, and retrofits it into a predicted pattern to re-run analytics and detect anomalies accurately.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the anomaly detection model uses available data after connectivity restoration, then the model can operate continuously, but the prediction accuracy deteriorates due to data outages and false positives from data surges
Solution Approach 1:
The system performs preliminary actions by detecting data outages in real-time, marking affected time periods, and generating synthetic data to fill gaps before anomaly detection is performed. This ensures that when the anomaly detection model runs continuously, it operates on complete data sets rather than fragmented post-restoration data, maintaining prediction accuracy while enabling continuous operation.
2Loss of information
If the model receives aggregated data during data outage, then data availability is maintained, but the pattern information is lost leading to inaccurate anomaly detection
Solution Approach 1:
The system creates a copy of the expected data pattern during outages by using predictive modeling to generate synthetic data that mimics what the actual data would have looked like. This synthetic copy preserves the temporal and contextual patterns that would otherwise be lost in aggregated data, enabling accurate anomaly detection even when real data is unavailable.
Solution Approach 2:
The system changes the parameter of data granularity by generating synthetic data at the original high-frequency sampling rate rather than accepting aggregated low-frequency data. This maintains the temporal resolution and pattern information necessary for accurate anomaly detection while ensuring data availability during outages.
3Reliability
If connectivity is restored after extended outage, then data flow resumes, but data surge causes false anomaly identification
Solution Approach 1:
The system performs preliminary actions by identifying and marking data outage periods before connectivity restoration occurs. When data resumes flowing, the system recognizes the previously marked outage periods and excludes the resulting data surge from anomaly detection, preventing false positives while maintaining reliable operation after connectivity restoration.
4Measurement precision
If the anomaly detection model processes data at high granularity, then detection precision is improved, but data requirements increase making the system more vulnerable to outages
Solution Approach 1:
The system creates synthetic copies of high-granularity data during outages using predictive modeling, maintaining both the precision benefits of high-granularity processing and the robustness against outages. The synthetic data fills gaps at the same temporal resolution as the original high-granularity data stream.
Data Source
AI summary
A computer-implemented method, system and computer program product for identifying anomalies in data during a data outage. An anomaly detection model is built using data received from a sensor at a characterized granularity. Once a period of service occurs following a data outage, a quantum of missing data during the data outage is identified using predictive modeling if the data during the data outage is not available at the granularity in which the anomaly detection model is built. The identified quantum of missing data is retrofitted into a predicted pattern during the data outage and the analytics are then re-run on the retrofitted quantum of missing data in the predicted pattern to identify anomalies during the data outage. In this manner, anomalies in data, such as data from sensor readings, can be identified during the data outage thereby enabling the model to provide more accurate predictions of anomalies occurring during the data outage.