Kalman Filter Anomaly Detection for Cloud VM Resilience
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection methods in cloud services are inadequate for real-time, automated identification of anomalies across multiple virtual machines, as they do not recursively adapt input gains and noise covariances, and lack robustness in log likelihood analysis, which hampers efficient migration and resilience of cloud services.
Innovation Solution
An ongoing anomaly detection system utilizing a modified Kalman Filter that recursively adapts input gains and noise covariances, combined with a moving average filter on log likelihoods, to detect anomalies in groups of virtual machines, triggering automatic live migration and enhancing cyber analyst focus.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional anomaly detection methods are used in cloud services, then implementation is simpler, but real-time automated detection across multiple virtual machines is inadequate and lacks adaptability
Solution Approach 1:
The system dynamically adapts the Kalman Filter parameters (input gains and noise covariances) in real-time based on observed data characteristics. This allows the anomaly detection system to adjust its sensitivity and response to changing conditions in the cloud environment, improving adaptability without requiring complete redesign of the detection architecture.
Solution Approach 2:
The patent changes the parameters of the Kalman Filter (specifically input gains and noise covariances) to optimize anomaly detection performance. By recursively adapting these parameters based on historical data and current observations, the system achieves better detection accuracy while maintaining a relatively simple filter-based architecture.
2Extent of automation
If basic Kalman Filter is used for anomaly detection, then implementation is simpler, but recursive adaptation of input gains and noise covariances is not achieved
Solution Approach 1:
The anomaly detection system performs self-adjustment by automatically adapting its own parameters (input gains and noise covariances) based on the data it processes. This self-service capability enables the system to improve its detection performance over time without external intervention, achieving higher automation while the added complexity is confined to the parameter adaptation logic.
3Reliability
If moving average filter on log likelihoods is not applied, then processing is faster, but robustness of anomaly indicator is reduced
Solution Approach 1:
The system applies a moving average filter to the log likelihoods as a preliminary processing step before making anomaly detection decisions. This preprocessing action smooths out fluctuations and provides a more robust anomaly indicator, reducing false positives while the computational overhead is manageable through efficient implementation.
Data Source
AI summary
A system and method that automatically detects anomalies in a cloud service system on an ongoing basis and which can be used to trigger live migration of cloud services includes a cloud server system configured to provide a plurality of virtualized cloud services through processes running over a set of virtual machines hosted on the cloud server system, and a processor which receives data related to the operations of the virtual machines and determines whether any of the virtual machines are exhibiting anomalous behavior. The processor applies a Kalman Filter to make predictions on the future state and covariance of the virtual machines and then calculates the log likelihood of the predicted values. If the predicted values deemed to be very unlikely, then the processor signals that an anomaly has occurred.