Duplicate Process for Low Latency Data Stream Snapshot

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current data stream processing systems fail to provide extremely low latency while maintaining the 'exactly-once' processing property, as they interrupt data stream processing to create snapshots for failure recovery, leading to incorrect final processing results.

Innovation Solution

A method is introduced where a duplicate process is activated upon receiving a trigger tag in one input data stream, allowing it to process data units before a snapshot tag in another input data stream without interrupting the main processing, and a snapshot is created upon receiving the snapshot tag for failure recovery, ensuring that data units are processed exactly once.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a snapshot is created for failure recovery in traditional data stream processing, then reliability is improved, but processing continuity deteriorates causing interruptions and increased latency

Engineering Contradiction:

Improvefailure recovery capabilityVSAvoidprocessing latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system segments the snapshot creation process into two independent parts: the main processing stream continues uninterrupted while a duplicate process handles snapshot creation separately. This segmentation allows the main stream to maintain low latency while the duplicate process ensures reliability through snapshot creation.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary action by creating a duplicate process in advance that mirrors the main processing stream. This duplicate process is prepared to take over snapshot creation duties, allowing the main stream to continue processing without interruption while the duplicate handles the reliability-critical snapshot operations.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If data stream processing is interrupted to create snapshots, then reliability is improved, but productivity deteriorates due to processing pauses

Engineering Contradiction:

Improveexactly-once processing guaranteeVSAvoiddata processing throughput

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The processing system is segmented into a main processing stream that maintains high throughput and a duplicate process that handles snapshot creation. This segmentation ensures that the main stream's productivity is not affected by the reliability-critical snapshot operations performed by the duplicate process.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system creates a copy (duplicate process) of the main processing stream specifically for handling snapshot creation. This copying allows the original main stream to continue processing data at full speed while the duplicate process independently manages snapshot operations to ensure exactly-once processing reliability.

Inventive Principle:

Principle #26Copying

3Device complexity

If the main process creates snapshots directly, then device complexity is reduced, but reliability deteriorates due to incorrect final processing results

Engineering Contradiction:

Improveprocess structure simplicityVSAvoidprocessing result accuracy

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The system segments the snapshot creation function from the main processing stream and assigns it to a duplicate process. This segmentation resolves the contradiction by maintaining the simplicity of the main process while ensuring reliability through the specialized duplicate process that handles snapshot operations independently.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The duplicate process acts as an intermediary between the main processing stream and the snapshot creation mechanism. This intermediary handles the reliability-critical snapshot operations, allowing the main process to remain simple while ensuring correct final processing results through the duplicate's dedicated snapshot management.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11157178B2Method, device and computer program product for data stream processing

Publication Date: 2021.10.26 EMC IP HLDG CO LLC

AI summary

Embodiments of the present disclosure relate to a method, device and computer program product for data stream processing. The method comprises activating, in response to receiving a starting first predefined tag in a first input data stream from a first data source, a duplicate process of a process by duplicating a state of the process to the duplicate process. The method further comprises processing, by the duplicate process, only a data unit before an ending second predefined tag in a second input data stream from a second data source without interruption of processing of the first and second input data streams by the process. The second data source differs from the first data source. The method further comprises creating a snapshot of the duplicate process for recovery of the process due to failure in response to receiving the ending second predefined tag in the second input data stream.