Stream Conditioning for Distributed Event Ordering

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed event processing systems face challenges in maintaining event order and timing accuracy due to latency and disorder in best effort networks, which can lead to incorrect answers and blocked queries, especially when events from different sources arrive at different times or are observed with latency.

Innovation Solution

The system employs a stream conditioning mechanism using 'punctuation' and 'heartbeats' to align events across multiple streams, establishing a logical clock based on physical clocks from multiple sources, and utilizes AI/MLR for probabilistic analysis to synchronize clocks and order events, thereby masking latency and disorder.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If events are processed in arrival order from distributed streams, then the system can operate without globally synchronized clocks, but the processing may produce incorrect answers due to latency and disorder

Engineering Contradiction:

Improveoperation without synchronized clocksVSAvoidcorrectness of processing results

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system performs preliminary ordering and timestamp assignment to events as they arrive, organizing them into candidate sets before final processing. This preliminary action ensures that even though events arrive out of order due to network latency, they are pre-organized according to their logical occurrence time, preventing incorrect processing results

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an intermediary event ordering mechanism that acts as a mediator between distributed event sources and the processing system. This intermediary assigns logical timestamps and organizes events into ordered sequences, resolving the conflict between operating without synchronized clocks and maintaining processing correctness

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If the system waits for events to arrive within fixed time intervals, then temporal correlation queries can be accurate, but queries may block indefinitely if streams are slow

Engineering Contradiction:

Improveaccuracy of temporal correlationVSAvoidquery execution speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system dynamically adjusts its event collection and processing behavior based on arrival patterns. Instead of rigidly waiting for fixed time intervals, the system adapts by using watermark-based progression and dynamic time windows, allowing queries to progress when sufficient events have arrived while maintaining accuracy through conditional continuation

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system performs preliminary organization of events into time-ordered sequences using logical timestamps before executing temporal correlation queries. This preliminary ordering allows the system to accurately evaluate temporal relationships without blocking, as events are already arranged in their logical occurrence order

Inventive Principle:

Principle #10Preliminary action

3Adaptability or versatility

If the system reasons over event arrival times, then it can handle out-of-order events, but it cannot determine the actual occurrence order without synchronized clocks

Engineering Contradiction:

Improvehandling of out-of-order eventsVSAvoidaccuracy of occurrence timing

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The system changes the time parameter representation from physical clock timestamps (which require synchronization) to logical timestamps derived from event sequence numbers and arrival patterns. This parameter transformation allows the system to reason about event ordering with precision even without synchronized clocks, as logical timestamps are computed based on the actual sequence and timing of event arrivals

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS8099452B2Event stream conditioning

Publication Date: 2012.01.17 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

A distributed event processing system that can organize (e.g., order) input streams regardless of actual time of receipt is provided. This order may simply be arrival order or given explicitly on a specific event attribute, such as a timestamp or sequence number. Traditional standing event queries, such as a temporal correlation of events across multiple streams, often block if an input stream is slow or may produce an incorrect answer if events fail to arrive within the fixed time interval. In response to this situation, the subject innovation employs punctuation and heartbeats in connection with a robust and scalable event processing system. The innovation discloses mechanisms by which heartbeats and timestamps can be regularly generated by low-level nodes (e.g., sources) and propagated through the network, to unblock standing event pattern queries and align events from multiple distributed streams.