Autoencoder Metrics Compression for Cloud Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of managing large volumes of metrics data generated by cloud resources is complex, requiring efficient storage and analysis to understand historical behavior and detect anomalies, which is hindered by high storage needs and manual analysis impracticality.
Innovation Solution
An autoencoder is used to encode and decode metrics data streams, generating reduced size data streams for efficient storage while detecting anomalies by comparing reconstruction errors and z-scores, allowing for automated anomaly detection and historical data retention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large volume of metrics data is stored to understand historical behavior, then anomaly detection capability is improved, but storage requirements and system complexity increase
Solution Approach 1:
The patent extracts only the essential information from the original metrics data by using an autoencoder to learn a compressed representation. The encoder captures the most important patterns and anomalies while discarding redundant information, thereby reducing storage requirements while maintaining anomaly detection capability.
Solution Approach 2:
The patent transforms the data from its original high-dimensional form to a lower-dimensional compressed representation by changing the parameter space. The autoencoder maps the original metrics data to a reduced feature space that preserves anomaly information while significantly reducing the volume of data that needs to be stored.
2Measurement precision
If manual analysis of metrics data is performed to detect anomalies, then detection precision is improved, but time consumption and operational complexity increase
Solution Approach 1:
The patent implements self-service anomaly detection where the autoencoder automatically identifies anomalies without human intervention. The system monitors itself by comparing the compressed representation against learned patterns, automatically flagging anomalies for further investigation, thereby eliminating manual analysis time while maintaining high detection precision.
Solution Approach 2:
The patent uses feedback mechanisms where the autoencoder continuously learns from the data patterns and updates its internal model. The reconstructed data is fed back into the system for comparison, allowing the system to automatically adjust its anomaly detection thresholds and improve precision over time without manual reconfiguration.
3Loss of information
If raw metrics data is stored for historical analysis, then data availability is improved, but storage cost and system resource consumption increase
Solution Approach 1:
The patent creates a compressed copy of the original metrics data through the autoencoder's encoder. This compressed representation preserves the essential information and anomaly patterns while occupying significantly less storage space. The compressed data serves as an efficient copy that maintains data availability for historical analysis while reducing storage costs.
Data Source
AI summary
Techniques for autoencoder based anomaly detection and efficient storage of metrics within a cloud environment are disclosed. In an example, an input data stream is received from a cloud resource operating within a cloud environment, input data stream indicative of a metric associated with the cloud resource. An autoencoder (i) encodes the input data stream to generate a reduced size data stream, (ii) decodes the reduced size data stream to generate an output data stream that is an estimated reconstruction of the input data stream, (iii) compares the input and output data stream, to generate a stream of reconstruction errors, and (iv) generates a stream of z scores, based on the stream of reconstruction errors. In an example, one or more data points within the input data stream are flagged as being anomalous data points, based at least in part on the stream of z scores.