Tail-Based Span Data Sampling for Distributed Traces
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for monitoring computing systems' performance through span data collection are inefficient, as they rely on random sampling, resulting in high volumes of data and increased operational costs due to unnecessary data transmission.
Innovation Solution
Implementing tail-based sampling at the trace level, where a trace manager groups operations into traces, compares trace characteristics with predefined criteria, and discards unselected data, reducing the volume of data transmitted while maintaining effective performance monitoring.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If head-based random sampling is used to reduce span data volume, then data transmission volume is reduced, but sampling intelligence and performance monitoring effectiveness deteriorate
Solution Approach 1:
The patent applies preliminary action by evaluating trace characteristics before finalizing sampling decisions. The system pre-processes span data to identify trace-level patterns and characteristics, then uses this preliminary analysis to make intelligent sampling decisions that preserve performance monitoring effectiveness while reducing data volume.
Solution Approach 2:
The patent inverts the traditional head-based sampling approach by implementing tail-based sampling. Instead of randomly selecting spans at the beginning of data collection, the system collects complete traces first and then selectively samples based on trace characteristics, reversing the conventional sampling timing and level to improve sampling intelligence.
2Measurement precision
If complete span data is collected for performance monitoring, then monitoring precision is improved, but data transmission costs and processing overhead increase
Solution Approach 1:
The patent extracts only the essential trace-level information needed for performance monitoring rather than transmitting complete span data. By identifying and extracting key trace characteristics, the system maintains monitoring precision while significantly reducing data transmission costs and processing overhead.
Solution Approach 2:
The patent applies partial action by selectively sampling traces based on their characteristics rather than processing all span data. The system performs partial sampling at the trace level, which is sufficient for effective performance monitoring without the excessive cost of transmitting and processing complete data sets.
3Quantity of substance
If random head-based sampling is applied at agent level, then data volume is reduced, but sampling intelligence and trace-level analysis capability are lost
Solution Approach 1:
The patent merges span data at the trace level before sampling decisions are made. By combining related spans into complete traces, the system preserves trace-level analysis capability and sampling intelligence while still achieving data volume reduction through selective trace sampling rather than random span selection.
Data Source
AI summary
A system provides for tail-based sampling of span data at the trace level. The system may include a circuitry, such as one or more servers, that execute applications, agents, and a trace manager. The agents generate span data of operations performed by the applications. The trace manager receives the span data from the agents, and groups the operations defined by the span data into traces. The trace manager determines a selected subset of the traces for span data sampling and an unselected subset of the traces excluded from the span data sampling by comparing one or more trace characteristics of the traces with one or more trace criteria. The trace manager discards unselected span data of the unselected subset of the traces and provide selected span data of the selected subset of the traces to a monitoring system via a network.