Microservices Monitoring Data Segmentation and Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional monitoring systems face challenges in efficiently ingesting and analyzing vast amounts of span and trace data from microservices-based applications, often resorting to data sampling which results in loss of information and inaccurate metric calculations.
Innovation Solution
The system ingests up to 100% of spans and converts them into metric data streams and traces using advanced compression methods, allowing for real-time monitoring and accurate calculation of high-cardinality metrics like throughput, latency, and error rates without sampling, and supports multiple modalities of analysis including metric time series, metric events, and full-fidelity modes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data sampling is used to reduce data volume, then processing speed and resource consumption improve, but measurement precision and information completeness deteriorate
Solution Approach 1:
The patent segments the vast machine data into structured, semi-structured, and unstructured components, and further divides it into event data and metric data. This segmentation allows selective processing and storage of different data types in appropriate formats, enabling efficient querying without processing the entire data set, thus resolving the contradiction between processing speed and measurement precision.
Solution Approach 2:
The patent introduces a new dimension of data representation by converting traditional time-series metrics into event-based data models with multiple dimensions (event type, source, timestamp, metrics). This dimensional transformation enables flexible querying and analysis without requiring full data processing, improving both processing efficiency and measurement accuracy.
2Loss of information
If all machine data is stored and processed, then measurement precision and information completeness improve, but device complexity and processing resources worsen
Solution Approach 1:
The patent applies local quality by storing different types of data in different formats and locations within the data system. Structured data is stored in relational databases, semi-structured data in NoSQL databases, and unstructured data in object storage. This localized optimization reduces overall system complexity while maintaining complete data availability for analysis.
Solution Approach 2:
The patent performs preliminary actions by pre-processing machine data during ingestion, converting raw data into structured event models with standardized schemas. This preliminary structuring enables efficient querying and analysis later without requiring complex processing of raw data, reducing system complexity while preserving data completeness.
3Speed
If data preprocessing is performed to extract specified items, then retrieval efficiency improves, but loss of information increases due to discarding remainder data
Solution Approach 1:
The patent creates a universal data model that serves multiple functions simultaneously. The event-based data structure supports both efficient querying for specific metrics and comprehensive analysis of all data. By designing a flexible schema that can accommodate different query types, the system achieves fast retrieval for common queries while preserving access to all data for deeper analysis, eliminating the need to discard remainder data.
Data Source
AI summary
A system can display content generated from one type of machine-generated data to a user via a graphical user interface. Based on an interaction with the machine-generated data, the system can determine an entity identifier associated with the machine-generated data and determine an entity type for the entity identifier. The system can map the entity type to one or more content generators associated with the entity type and communicate the entity identifier to the identified content generators. The content generators can determine if they have content associated with the machine-generated data. The system can generate and display a link to the related content via a graphical user interface.