Data Stream Processor Transition System Using Timestamp Coordination

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data stream processors face challenges in transitioning control between old and new processors, leading to latency, errors, and out-of-order event processing due to difficulties in retrieving and synchronizing state information, which results in gaps or duplication of information in publish-subscribe environments.

Innovation Solution

A data stream processors transition system that uses timestamp information to coordinate the transfer of control between data stream processors, ensuring seamless operation by configuring time ranges for both processors to cover all necessary points in time, thereby preventing gaps or duplication of processed event results.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If state information is stored and transferred between data stream processors, then control can be transitioned between processors, but latency and errors increase during the transition process

Engineering Contradiction:

Improvecontrol transition capabilityVSAvoidtransition accuracy

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system performs preliminary actions by having the new data stream processor join the event stream and synchronize state information before the old processor stops. This includes the new processor receiving events, applying state transitions, and publishing results in advance, so that when control transitions, the state is already synchronized and no additional latency or errors occur during the handoff process.

Inventive Principle:

Principle #10Preliminary action

2Adaptability or versatility

If the existing data stream processor is stopped to transfer control, then state information can be retrieved and stored, but processing latency increases and errors may occur

Engineering Contradiction:

Improveprocessor replacement capabilityVSAvoidtransition latency

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system maintains continuity of useful action by having the new data stream processor join the event stream and begin processing events before the old processor stops. The new processor continuously receives events, applies state transitions using synchronized state information, and publishes results, ensuring that processing continues without interruption and eliminating transition latency.

Inventive Principle:

Principle #20Continuity of useful action

3Productivity

If events are processed out of chronological order during transition, then processing can continue, but event integrity is compromised and results may be incorrect

Engineering Contradiction:

Improvecontinuous processingVSAvoidevent processing accuracy

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The system uses an intermediary mechanism where the new data stream processor receives a copy of the event stream and synchronizes state information from the old processor. This intermediary state synchronization ensures that both processors work with consistent, chronologically ordered events, maintaining event integrity while allowing continuous processing without interruption.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS10489219B2Transitioning between data stream processors in a publish-subscribe system

Publication Date: 2019.11.26 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

A data stream processors transition system which implements techniques for transitioning control from a first data stream processor to a second data stream processor. The data stream processors transition system preserves the integrity of data streams produced by first and second data stream processors when transitioning control between the first to the second data stream processors. For example, timestamp information may be provided to the first data stream processor that specifies when the first data stream processor may transfer control to the second data stream processor. The first data stream processor may process events by consuming and publishing results until the time specified by the timestamp information. The second data stream processor may, after the time specified by the timestamp information, process events by consuming and publishing results.