Distributed Trace Aggregation for System Behavior Visibility

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current distributed tracing technologies in online distributed computer systems lack a holistic and comprehensive view of system behavior and are unable to perform targeted tracing, making it difficult for developers and administrators to diagnose performance issues and troubleshoot errors effectively.

Innovation Solution

The implementation of distributed trace aggregation and targeted tracing methods, which involve generating trace events across multiple system nodes, collecting and aggregating them, and enabling precise tracing of specific user requests through edge nodes, allowing for comprehensive system insights and diagnostic capabilities.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If distributed tracing is implemented across thousands of computing devices, then system behavior visibility is improved, but system complexity and resource consumption increase

Engineering Contradiction:

Improvesystem behavior visibilityVSAvoidtracing system complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The tracing system is segmented into distributed sampling agents on individual computing devices that independently generate trace events, and a centralized aggregation service that collects and processes these events. This segmentation allows local autonomy while achieving global visibility, reducing the complexity burden on any single component.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A trace aggregation service acts as an intermediary between distributed tracing sources and analysis tools. This mediator collects trace events from multiple devices, performs aggregation and filtering, and presents consolidated information, thereby simplifying the overall system architecture and reducing direct complexity at each endpoint.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If all user requests are traced, then diagnostic capability is improved, but resource consumption and processing overhead increase

Engineering Contradiction:

Improvediagnostic capabilityVSAvoidtracing resource consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

Instead of tracing all user requests, the system implements probabilistic sampling where only a subset of requests is traced based on configurable sampling rates. This partial action approach provides sufficient diagnostic capability for most scenarios while significantly reducing resource consumption. The sampling rate can be adjusted to balance diagnostic needs against resource constraints.

Inventive Principle:

Principle #16Partial or excessive action

3Loss of information

If comprehensive trace data is collected, then system insight is improved, but data processing and storage requirements increase

Engineering Contradiction:

Improvesystem insightVSAvoidtrace data volume

Core Design Contradiction:

Loss of informationVSQuantity of substance

Solution Approach 1:

The system extracts only the most relevant trace events and metrics from the full trace data stream. By filtering and selecting specific information that provides the most insight into system behavior and performance, the system maintains comprehensive understanding while significantly reducing the volume of data that needs to be processed and stored.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

Trace events from multiple computing devices and services are merged and aggregated into consolidated views. This combining of data from distributed sources into unified metrics and summaries reduces the total data volume while preserving comprehensive system insight, as the aggregation process synthesizes information rather than simply accumulating raw data.

Inventive Principle:

Principle #5Merging (Combining)

4Productivity

If targeted tracing of specific user requests is enabled, then troubleshooting efficiency is improved, but tracing system flexibility is reduced

Engineering Contradiction:

Improvetroubleshooting efficiencyVSAvoidtracing flexibility

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The tracing system implements dynamic configuration where sampling rates, filtering criteria, and target selection can be adjusted in real-time based on system conditions and diagnostic needs. This dynamic adaptability allows the system to switch between comprehensive monitoring and targeted troubleshooting modes, maintaining both troubleshooting efficiency and system flexibility.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS20210119892A1Online computer system with methodologies for distributed trace aggregation and for targeted distributed tracing

Publication Date: 2021.04.22 NETFLIX INC

AI summary

An online distributed computer system with methodologies for distributed trace aggregation and targeting distributed tracing. In one aspect, the disclosed distributed tracing technologies improve on existing distributed tracing technologies by providing to application developers and site operations personnel a more holistic and comprehensive insight into the behavior of the online distributed computer system in the form of computed span metric aggregates displayed in a graphical user interface thereby making it easier for such personnel to diagnose problems in the system and to support and maintain the system. In another aspect, the disclosed distributed tracing technologies improve on existing distributed tracing technologies by facilitating targeted tracing of initiator requests.