Aggregated Data Models for Mass Metric Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale data processing systems generate vast amounts of log data, making it difficult to store and diagnose issues due to the massive quantity of metrics produced, which increases storage needs and complicates problem-solving.
Innovation Solution
Implementing a data model aggregation system where instead of storing all metrics, aggregated data models are created, reducing storage space and allowing for monitoring of system health without storing original metrics, with control over data storage retained by the source system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all metrics are stored for monitoring purposes, then complete diagnostic information is available, but storage requirements become prohibitively large
Solution Approach 1:
The patent extracts only the essential diagnostic information from the complete metric data set. Instead of storing all raw metrics, the system identifies and stores only the key performance indicators and anomaly detection data needed for effective monitoring, thereby reducing storage requirements while maintaining diagnostic capability
Solution Approach 2:
The patent inverts the traditional approach by not storing the original metrics directly, but rather storing processed and aggregated data models that represent the essential information. The system transforms the data representation from raw metrics to compressed data models that capture diagnostic essence without requiring full metric storage
2Quantity of substance
If aggregated data models are used instead of original metrics, then storage requirements are reduced, but data flexibility and control are lost
Solution Approach 1:
The patent implements dynamic dimension management where the data model structure can be adjusted and modified based on monitoring needs. The system allows for flexible configuration of data dimensions and aggregation parameters, enabling adaptation to different diagnostic requirements while maintaining storage efficiency
Solution Approach 2:
The patent segments the data storage into multiple dimensions and hierarchical levels. The data model is divided into configurable dimensions that can be independently managed, allowing the source system to retain control over specific data aspects while benefiting from overall storage reduction through aggregation
Data Source
AI summary
Disclosed are various embodiments for processing and storing mass data, where the data may include metrics generated based on performance of an event in a monitored system. Metrics describing a state of a monitored system may be received, accessed, and aggregated to generate a data model that describes performance of the monitored system. The metrics utilized in generating the data model may be disregarded after the data model has been generated. An output describing the state of the monitored system may be generated based on the data model, and the output may be communicated over a network, for example, to a requesting service.