Microservice Trace Storage Modes for Accurate Queryable Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional monitoring tools for microservices architectures struggle to efficiently ingest and aggregate significant amounts of trace data, leading to data loss and inaccurate performance calculations due to exclusive data sampling.
Innovation Solution
A monitoring platform that ingests, processes, and stores all incoming spans in real time, using advanced compression methods, and supports multiple analysis modes: fast and full-fidelity, fast and approximate, and slow and full-fidelity, to provide accurate metrics on throughput, latency, and error rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional monitoring tools use data sampling to reduce data volume, then storage and processing requirements are reduced, but measurement precision and reliability of performance metrics deteriorate
Solution Approach 1:
The patent segments trace data into different storage modes: hot storage for recently ingested spans enabling full-fidelity analysis, and cold storage for aggregated/span-level data enabling approximate analysis. This segmentation allows the system to maintain high measurement precision for recent data while reducing storage requirements through aggregation for historical data.
Solution Approach 2:
The patent changes the parameter of data representation from individual span-level detail to aggregated metrics (count, min, max, sum) when storing in cold storage. This parameter transformation reduces data volume while preserving sufficient information for approximate performance analysis, resolving the contradiction between data volume and measurement precision.
2Reliability
If all incoming spans are ingested and stored in real time, then measurement precision and reliability improve, but device complexity and processing requirements increase
Solution Approach 1:
The patent divides the storage system into hot storage and cold storage components, each with different processing requirements. Hot storage maintains full span fidelity for accurate analysis, while cold storage uses aggregated data for approximate analysis. This segmentation reduces overall system complexity by distributing different processing demands across separate storage layers.
Solution Approach 2:
The patent implements partial ingestion where spans are ingested in full detail into hot storage, but only aggregated metrics are maintained in cold storage. This partial action approach provides sufficient reliability for most analysis scenarios while avoiding the excessive complexity of maintaining full fidelity across all storage.
3Adaptability or versatility
If multiple analysis modes are supported simultaneously, then adaptability and versatility improve, but device complexity increases
Solution Approach 1:
The patent creates a universal storage architecture that supports multiple analysis modes (full-fidelity and approximate) through a unified hot/cold storage framework. This multi-functional design allows the same system to serve different query types without requiring separate specialized systems, improving adaptability while managing complexity through a standardized interface.
Solution Approach 2:
The patent segments the storage system into hot and cold layers that can independently support different analysis modes. This segmentation allows flexible combination of storage modes to support multiple analysis types simultaneously, achieving versatility while keeping each segment's complexity manageable.
4Productivity
If data is aggregated for faster analysis, then productivity and response time improve, but measurement precision deteriorates
Solution Approach 1:
The patent segments data storage into hot storage with full span detail for precise analysis and cold storage with aggregated metrics for fast analysis. This segmentation allows the system to provide both high precision and high speed analysis by routing queries to appropriate storage layers based on requirements.
Solution Approach 2:
The patent transforms detailed span data into aggregated parameters (count, min, max, sum) for cold storage, enabling fast mathematical computations. This parameter change allows productivity improvement through efficient aggregation while maintaining the option to retrieve original spans from hot storage when precision is critical.
Data Source
AI summary
A method of analyzing a performance of a microservices-based application comprises generating a plurality of traces from a plurality of spans associated with the microservices-based application. The method also comprises generating a plurality of data sets each associated with a respective analysis mode of a plurality of analysis modes using the plurality of traces, wherein each analysis mode extracts a different level of detail for analyzing the performance of the services in the application from the plurality of spans. Further, the method comprises selecting, based on a first user query, a first analysis mode from the plurality of analysis modes for generating a response to the first user query. The method also comprises accessing a data set of the plurality of data sets that is associated with the first analysis mode and generating the response to the first user query using the data set associated with the first analysis mode.