Stream Processing Diagnostics via Indexed Activity Records

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed stream-processing systems face challenges in detecting, diagnosing, and resolving errors due to the lack of detailed information about hosts or components experiencing issues, leading to time-consuming manual diagnosis and increased latency or downtime, especially when stateful processing involves maintaining data across multiple events and streams.

Innovation Solution

A diagnostics system that periodically collects and indexes activity records from subcomponents of each job in a stream-processing system, generating a diagnostics event stream with key-value pairs for efficient error tracking and root cause analysis, allowing administrators to analyze performance and take remedial actions automatically.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If conventional monitoring systems are used to detect errors in distributed stream-processing systems, then error detection capability is provided, but detailed information about hosts or components experiencing errors is lost, requiring time-consuming manual diagnosis

Engineering Contradiction:

Improveinformation about hosts or components experiencing errorsVSAvoidtime for manual diagnosis

Core Design Contradiction:

Loss of informationVSLoss of time

Solution Approach 1:

The system performs preliminary actions by continuously collecting and indexing diagnostic information about all hosts and components before errors occur. This pre-collected information is stored in an optimized format that enables immediate retrieval when errors happen, eliminating the need for manual diagnosis and reducing both information loss and time loss.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

An intermediary diagnostic information collection and indexing system is introduced between the stream-processing components and the monitoring system. This intermediary continuously gathers detailed diagnostic data from subcomponents and organizes it in advance, so when errors occur, the information is already prepared and readily available, preventing information loss and reducing diagnosis time.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If stateful stream processing is implemented to maintain data across multiple events and streams, then processing capability is improved, but recovery time increases significantly when node failures occur

Engineering Contradiction:

Improvestream processing capabilityVSAvoidnode recovery time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates and maintains copies of diagnostic information and state data across multiple nodes and storage locations. When a node fails, these pre-existing copies can be immediately utilized for recovery, eliminating the need to replay processing from scratch and significantly reducing recovery time while maintaining processing capability.

Inventive Principle:

Principle #26Copying

3Measurement precision

If diagnostic information is collected and indexed for all jobs and subcomponents, then error analysis capability is improved, but processing and network overhead increases

Engineering Contradiction:

Improveerror tracking precisionVSAvoidprocessing and network overhead

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The system applies local quality by collecting and indexing diagnostic information at the appropriate level of granularity for each subcomponent and job. Rather than uniformly collecting all possible data everywhere, the system tailors the collection scope to local needs, gathering detailed information where errors are most likely to occur and less detailed information elsewhere, thus improving error tracking precision while minimizing processing and network overhead.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system performs partial action by selectively collecting and indexing only the most relevant diagnostic information needed for effective error analysis. Rather than collecting all possible data, the system focuses on key metrics and events that provide the highest value for error tracking, reducing processing and network overhead while maintaining sufficient measurement precision for effective diagnostics.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11308092B2Stream processing diagnostics

Publication Date: 2022.04.19 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

The disclosed embodiments provide a system for processing data. During operation, the system receives records of activity within a stream-processing system over a set of event streams, wherein each event stream in the set of event streams contains events related to a corresponding job in the stream-processing system. Next, the system indexes data in the records under a set of keys that include a first key related to jobs in the stream-processing system and a second key related to errors in the stream-processing system. The system then outputs the indexed data for use in analyzing the execution of the stream-processing system.