Distributed Tracing with Head-Based and Rate-Limited Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed tracing methods, such as head-based, tail-based, and rate-limiting sampling, have limitations that result in inefficient resource usage and incomplete data collection in microservice architectures, leading to unmanageable tracing data volumes and missed information.
Innovation Solution
A method that combines head-based and rate-limiting sampling by identifying and retaining tracing data samples from preceding microservices, using an ingress gateway to set sampling rates and inserting labels, and utilizing tracing agents to ensure comprehensive data collection while filtering out unnecessary samples.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If head-based sampling is used to reduce tracing data volume, then resource consumption is reduced, but error information is lost because sampling decisions are made before errors occur
Solution Approach 1:
The system performs head-based sampling to preliminarily select traces for collection, reducing data volume early in the process. This preliminary action captures potential error traces while minimizing resource consumption by not collecting all traces.
Solution Approach 2:
The system implements feedback by analyzing trace results after completion and using this information to inform future sampling decisions. The error detection mechanism provides feedback that allows the system to identify which previously sampled traces contained errors and adjust sampling strategies accordingly.
2Loss of information
If tail-based sampling is used to capture error information, then comprehensive information is provided, but tracing data volume increases to an unmanageable size
Solution Approach 1:
The system performs preliminary head-based sampling to select a subset of traces for collection before they complete execution. This preliminary selection prevents the unmanageable data volume problem by avoiding collection of all traces, while still capturing error traces that occur in the sampled subset.
Solution Approach 2:
The system applies different sampling strategies to different portions of the trace data flow. Head-based sampling is applied to the incoming trace stream to reduce volume, while error detection mechanisms are applied locally to identified error traces to ensure comprehensive error information is captured without increasing overall volume.
3Productivity
If rate-limiting sampling is used to control sampling rate, then sampling rate is adjustable, but accuracy of sampling is reduced
Solution Approach 1:
The system performs head-based sampling as a preliminary action with predetermined sampling rates configured in the ingress gateway. This preliminary sampling establishes a controlled data flow that maintains accuracy while enabling rate adjustment through configuration parameters.
Solution Approach 2:
The system enables parameter changes by allowing configuration of sampling rates and error thresholds through the ingress gateway and tracing server. These parameter adjustments maintain sampling accuracy by using statistically sound sampling rates rather than arbitrary rate limiting.
Data Source
AI summary
Enhanced distributed tracing is provided. A set of microservices in a sequence of microservices preceding a preselected microservice in the sequence of microservices are identified. Each tracing data sample received from the set of microservices preceding the preselected microservice in the sequence of microservices is identified. Each tracing data sample corresponding to a set of selected transactions received from the set of microservices preceding the preselected microservice in the sequence of microservices that was dropped during rate-limiting sampling performed by the preselected microservice is deleted based on analysis of a result of the rate-limiting sampling of the set of selected transactions received from the preselected microservice thereby retaining a subset of tracing data samples corresponding to the set of selected transactions collected during head-based sampling.