Anomaly Database System for Sparse Telemetry Data Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing complexity and volume of telemetry data pose challenges in data management and storage, particularly in accessing reliable data backup and storage, and in fast data recovery in case of failure. Additionally, existing systems struggle with efficient anomaly detection and root cause analysis due to insufficient telemetric data.
Innovation Solution
An anomaly database system is introduced, which includes a continuous data management (CDM) node with a metrics library for sending system metrics in a sparse manner, a statistics relay for receiving streaming metrics, and a sparse consumers module that runs sparse algorithms to reduce data points. This system also features a rollup module for enabling read queries over designated time ranges and a baseline estimator for pre-computing baselines on streaming metrics to facilitate anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If telemetry data is collected and stored in full detail, then anomaly detection and root cause analysis are improved, but data management complexity and storage requirements increase rapidly
Solution Approach 1:
The patent extracts only the essential information from telemetry data by identifying and retaining only anomalous data points while discarding normal data. The system uses a database that stores only anomalies with their timestamps and values, extracting the critical information needed for detection while eliminating the bulk of redundant normal data, thus reducing storage requirements and management complexity.
Solution Approach 2:
The patent changes the parameter of data storage from storing all telemetry data points to storing only anomalous data points. By changing the selection criterion from complete data to filtered anomaly data, the system maintains detection reliability while significantly reducing the volume of data that needs to be managed and stored.
2Reliability
If telemetry data is stored in full detail, then root cause analysis is improved, but storage space and data recovery time increase
Solution Approach 1:
The system extracts and stores only anomalous data points rather than complete telemetry data. The database is designed to store only the essential anomaly information (timestamp, value, metadata) needed for root cause analysis, eliminating the need to store and manage the entire volume of normal operational data.
Solution Approach 2:
The patent changes the storage parameter from storing all data points to storing only anomalous data points. This parameter change reduces storage volume significantly while maintaining the capability to perform root cause analysis on the extracted anomaly data, which contains the critical information needed for diagnosis.
3Quantity of substance
If sparse algorithms are applied to reduce data points, then storage efficiency is improved, but data processing complexity increases
Solution Approach 1:
The system implements self-service anomaly detection where the sparse algorithm automatically identifies and extracts anomalous data points without requiring manual intervention. The database structure and retrieval mechanisms are pre-configured to handle sparse data efficiently, allowing the system to self-manage the complexity of filtering and storing only essential anomaly information.
Data Source
AI summary
In some examples, an anomaly database system is provided for processing metrics in telemetry data. An example anomaly database system comprises a continuous data management (CDM) node, the CDM node including a metrics library for sending out system metrics in a sparse manner and a statistics relay for receiving streaming metrics from nodes in a node cluster, the node cluster including the CDM node, the statistics relay pushing the received metrics to a metrics collector. A sparse consumers module pulls metrics, from the metrics collector, pushed to the metrics collector by the statistics relay.