Real-Time Data Stream Filter Generation for Microservices Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional monitoring systems face challenges in efficiently ingesting and analyzing vast amounts of span and trace data from microservices-based applications, often resorting to data sampling which results in loss of information and inaccurate metric calculations.
Innovation Solution
The proposed solution allows for the ingestion and analysis of up to 100% of spans, generating metric time series and supporting multiple modalities of analysis, including metric time series, metric events, and full-fidelity modes, to provide accurate and detailed insights into microservices-based applications without data loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data sampling is used to reduce data volume, then processing speed and resource consumption improve, but measurement precision and information completeness deteriorate
Solution Approach 1:
The system applies partial action by selectively processing only the necessary portions of data for specific analysis types. Different processing depths are applied based on the analysis modality required, avoiding unnecessary processing of all data while maintaining accuracy where needed.
Solution Approach 2:
The data processing system is segmented into multiple independent processing pipelines, each optimized for specific analysis modalities (metric time series, metric events, full-fidelity). This allows parallel processing of different data subsets with appropriate precision levels for each type of analysis.
2Measurement precision
If all span data is ingested and analyzed, then measurement precision and insight quality improve, but device complexity and processing overhead worsen
Solution Approach 1:
The system dynamically adjusts processing depth and data retention based on the type of analysis requested. Different modality modes (time series, events, full-fidelity) activate different processing pathways, allowing the system to maintain high precision when needed while reducing complexity for routine monitoring.
Solution Approach 2:
Different quality levels of data processing are applied to different parts of the system based on local requirements. Critical metrics requiring high precision use full-fidelity processing, while less critical monitoring uses aggregated time-series processing, optimizing the balance between accuracy and complexity.
3Productivity
If data preprocessing with extraction is applied, then retrieval efficiency improves, but information completeness and analysis flexibility worsen
Solution Approach 1:
The system maintains universal access to raw span data across all processing modalities, allowing the same data source to serve multiple analysis purposes. The architecture enables switching between different analysis approaches (aggregated metrics, detailed events, full-fidelity analysis) without data loss, preserving both efficiency and flexibility.
Solution Approach 2:
Data is preprocessed into multiple parallel formats and storage structures during ingestion, preparing it for various types of analysis simultaneously. This preliminary organization enables efficient retrieval for different analysis types without requiring reprocessing or data transformation later.
Data Source
AI summary
A data stream processing system can receive a stream of data and display a portion of the stream to a user. The displayed streaming data can change over time as additional data is received as part of the stream. The data stream processing system can extract one or more field values rom data in the stream and generate filters based on the extracted information. The generated filters can be displayed to a user, and in response to an interaction with a generated filter, the data stream processing system can apply the selected filter to data in the data stream.