Windowless Real-Time Join System for Data Stream Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional Complex Event Processing (CEP) systems delay join operations due to the elapsed time of windows such as batch, sliding, or hopping windows, even though they are described as real-time, which hinders immediate data processing.

Innovation Solution

Implementing a method for windowless real-time joins by initializing data structures and pre-computing steps before data receipt, allowing for immediate join updates as data arrives, using data models with transition stores and record comparators to manage join operations without delay.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If conventional CEP systems use batch, sliding, or hopping windows for processing, then processing can be performed in organized time intervals, but join operations are delayed by the window elapsed time

Engineering Contradiction:

Improvejoin operation delayVSAvoidprocessing speed

Core Design Contradiction:

Loss of timeVSProductivity

Solution Approach 1:

The patent pre-computes join steps and initializes data structures (hash maps, transition stores) before data receipt. This preliminary preparation eliminates the need to wait for window completion, as the join logic is already ready to execute immediately when data arrives, thus resolving the contradiction between time loss and productivity

Inventive Principle:

Principle #10Preliminary action

2Reliability

If window-based processing is used to organize data flow, then systematic processing is achieved, but real-time responsiveness is compromised

Engineering Contradiction:

Improveprocessing systematicityVSAvoidreal-time response speed

Core Design Contradiction:

ReliabilityVSSpeed

Solution Approach 1:

The patent transitions from static window-based processing to dynamic event-driven processing. The system dynamically responds to incoming events by executing pre-computed join steps immediately, while maintaining systematic processing through structured data models and transition stores. This dynamic approach resolves the contradiction between systematicity and response speed

Inventive Principle:

Principle #15Dynamics

3Measurement precision

If join operations wait for window completion, then accurate batch processing is ensured, but immediate data association is lost

Engineering Contradiction:

Improvebatch processing accuracyVSAvoiddata association delay

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent introduces transition stores as intermediary structures that bridge batch processing accuracy and real-time speed. These stores temporarily hold pre-computed join logic and data associations, allowing immediate data association while maintaining the systematic accuracy of batch processing through structured intermediate representations

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS9965517B2Windowless real-time joins

Publication Date: 2018.05.08 60EAST TECH

AI summary

A method for performing windowless real-time joins. The method can be implemented via execution of computer instructions configured to run at one or more processing modules and configured to be stored at one or more non-transitory memory storage modules. The method can include, prior to receiving data, initializing data structures. The method also can include, prior to receiving data, pre-computing steps to be performed upon receiving the data. The method additionally can include, upon receiving the data, performing windowless join updates. Other embodiments are provided.