Dynamic Trace Sampling for High-Latency Microservice Alerts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for analyzing instrumented software face challenges in efficiently communicating and storing spans and traces, as random sampling may miss valuable error traces amidst a majority of error-free traces, leading to unnecessary infrastructure costs and incomplete performance analysis.
Innovation Solution
The system annotates spans with tags to provide context, allowing up to 100% of incoming spans to be ingested and analyzed, using a latency analyzer to identify performance issues and an error analyzer to uncover error patterns, enabling detailed analysis and alert generation based on trace durations and tags.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If random sampling is used to transmit traces, then infrastructure costs are reduced, but valuable error traces may be missed
Solution Approach 1:
The system changes the sampling parameter from fixed random sampling to dynamic sampling based on trace characteristics. Traces are sampled based on parameters such as error presence, latency thresholds, and tag-based categorization, allowing the system to capture valuable error traces while reducing overall data transmission volume and infrastructure costs.
Solution Approach 2:
The sampling rate is made dynamic rather than static. The system adjusts sampling behavior based on real-time conditions, increasing sampling for error-prone or high-latency traces while maintaining lower sampling rates for normal traces, thereby optimizing both cost efficiency and information capture.
2Reliability
If all spans and traces are transmitted and stored, then complete analysis is achieved, but infrastructure costs increase significantly
Solution Approach 1:
The system extracts only the most valuable traces for transmission and storage based on predefined criteria such as error presence, latency thresholds, and tag matching. By taking out only the relevant subset of traces rather than all traces, the system achieves reliable analysis of critical issues while significantly reducing infrastructure costs associated with storing and processing all trace data.
Solution Approach 2:
Instead of transmitting all traces (excessive action), the system transmits a carefully selected partial set of traces that are most likely to contain valuable information for analysis. This partial action approach maintains analysis reliability for critical issues while reducing overall data volume and infrastructure requirements.
3Loss of energy
If traces are filtered to reduce data volume, then infrastructure costs are reduced, but valuable error traces may be lost
Solution Approach 1:
The system performs preliminary filtering and tagging of traces before transmission, marking traces with metadata that identifies their importance (e.g., error presence, latency issues, critical tags). This preliminary action enables the filtering process to preserve valuable error traces by identifying them in advance, while still reducing data volume for non-critical traces, thereby reducing infrastructure costs without losing important information.
Data Source
AI summary
A computer-implemented method for analyzing spans and traces associated with a microservices-based application executing in a distributed computing environment comprises aggregating a plurality of ingested spans associated with one or more applications executing in the distributed computing environment into a plurality of traces, wherein each of the plurality of ingested spans is associated with a plurality of tags. The method further comprises comparing durations of a set of related traces of the plurality of traces to determine patterns for the plurality of tags and generating a histogram that represents a distribution of the durations of the set of related traces. The method also comprises providing alerts for one or more tags from the plurality of tags associated with traces having a duration above a threshold based on the distribution of the durations.