Automated Feature Engineering for Streaming Data Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection systems for streaming data, particularly in industrial settings like refrigeration units, face challenges in distinguishing normal fluctuations from abnormal behavior due to varying operational patterns and lack of baseline data, leading to reactive rather than proactive detection methods.
Innovation Solution
An automated feature engineering method that converts streaming data into point statistics through long-term and short-term calculations, normalization, and dynamic time warping, enabling unsupervised anomaly detection algorithms like isolation forests and one-class SVM to identify anomalies before they occur.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional anomaly detection methods are used for streaming data, then the system can detect anomalies, but it reacts after anomalies occur rather than detecting them proactively
Solution Approach 1:
The system performs preliminary feature engineering by computing long-term and short-term point statistics from streaming data before anomaly occurrence. It calculates statistical features (mean, standard deviation, skewness, kurtosis) over different time windows in advance, creating a foundation for proactive anomaly detection rather than reactive response.
Solution Approach 2:
The system dynamically adjusts time windows and statistical calculations based on streaming data characteristics. It adapts the length of time windows and the granularity of feature extraction to match the operational patterns of the monitored system, enabling proactive detection of deviations from normal behavior.
2Loss of time
If the system processes streaming data in real-time, then it can detect anomalies promptly, but the complexity of feature engineering increases
Solution Approach 1:
The system segments the feature engineering process into distinct components: long-term statistics calculation, short-term statistics calculation, normalization, and anomaly detection. Each segment processes data independently at different time granularities, reducing the complexity burden on any single processing stage while maintaining real-time capability.
Solution Approach 2:
The system introduces intermediate statistical features (point statistics from long-term and short-term windows) as mediators between raw streaming data and the final anomaly detection algorithm. These intermediate features simplify the data representation and prepare it for efficient processing by anomaly detection models.
3Measurement precision
If the system uses multiple time windows for statistical calculations, then it can capture different temporal patterns, but the computational load increases
Solution Approach 1:
The system applies different time window lengths and statistical calculation methods to different aspects of the data. It uses longer time windows for capturing overall patterns and shorter time windows for detecting recent changes, with each window serving its specific local purpose in the overall anomaly detection strategy.
Solution Approach 2:
The system dynamically adjusts parameters such as time window lengths, statistical features to compute, and calculation granularities based on data characteristics and operational requirements. This allows optimization of computational energy consumption while maintaining measurement precision for different temporal patterns.
Data Source
AI summary
Feature engineering, for example, in automated machine learning, can include receiving streaming data representing at least one attribute detected by a sensor over time. Long term point statistics associated with the streaming data can be computed. The streaming data can be quantized into intervals of time windows and short term point statistics based on the intervals can be computed. The long term point statistics and the short term point statistics can be normalized. Dynamic time warping can be applied across the normalized long term point statistics and short term point statistics. A pair of probability distributions can be generated associated with the dynamic time warped normalized long term point statistics and short term point statistics. Based on distance between the mean values of the probability distributions, machine learning input features can be produced. The machine learning input features can be fed to train a machine learning model for detecting anomaly.