Real-Time Query Execution on Data Streams

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional data warehouse and SQL-based approaches fail to provide real-time data processing due to their inability to monitor continuous data streams and meet strict latency requirements for applications like fraud detection and real-time analytics, as they rely on historical data and ETL processes that are time-consuming.

Innovation Solution

A computer-implemented method and system that execute queries on near-real-time data streams, generating intermediate results as the query is executed, using dataflow graphs to process data in real-time, and aggregating results dynamically, allowing for user-defined operations and alerts, without the need for stored data access.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If traditional data warehouse and SQL-based approaches are used, then historical data can be processed, but real-time data processing capability is lost due to ETL time consumption and inability to monitor continuous data streams

Engineering Contradiction:

Improvedata processing speedVSAvoidETL processing time

Core Design Contradiction:

SpeedVSLoss of time

Solution Approach 1:

The patent performs data transformation and query preparation in advance, maintaining processed data in a ready-state for immediate querying. Data is transformed continuously as it arrives in the stream, eliminating the need for time-consuming ETL processes when queries are executed.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system continuously processes data streams in real-time, maintaining an ongoing state of data availability for querying. The data transformation and query execution occur continuously as data arrives, rather than in batch processes, ensuring constant readiness for real-time analytics.

Inventive Principle:

Principle #20Continuity of useful action

2Reliability

If data is collected, cleansed and integrated through ETL processes in a data warehouse, then data quality is improved, but latency requirements are not met due to the time required for these processes

Engineering Contradiction:

Improvedata qualityVSAvoidlatency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Data cleaning and transformation operations are performed preliminarily and continuously as data enters the stream, rather than in batch ETL processes. This preliminary processing maintains data quality while eliminating latency, as the data is ready for querying immediately upon arrival.

Inventive Principle:

Principle #10Preliminary action

3Loss of information

If SQL queries are executed on historical data in a data warehouse, then complete data analysis is achieved, but real-time intermediate results cannot be provided during query execution

Engineering Contradiction:

Improvequery result completenessVSAvoidquery execution time

Core Design Contradiction:

Loss of informationVSLoss of time

Solution Approach 1:

The system pre-computes and maintains intermediate query results continuously as data arrives in the stream. When a query is executed, the system can provide intermediate results immediately based on pre-computed data, while continuing to process incoming data to update results, thus providing both completeness and real-time feedback.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The query execution process continues continuously as new data arrives, with intermediate results being generated and updated in real-time. This continuous processing allows the system to provide progressively complete results during execution, rather than waiting for complete batch processing to finish.

Inventive Principle:

Principle #20Continuity of useful action

4Reliability

If traditional approaches are used to meet strict service level agreement requirements for real-time applications, then historical data analysis is possible, but real-time monitoring and pattern detection in continuous data streams is not achieved

Engineering Contradiction:

Improveservice level agreement complianceVSAvoidreal-time data access speed

Core Design Contradiction:

ReliabilityVSSpeed

Solution Approach 1:

The system continuously monitors data streams in real-time, maintaining ongoing query execution and pattern detection as data arrives. This continuous processing ensures service level agreement compliance for real-time applications while providing immediate data access speed, as the system is always ready to detect patterns and return results without batch processing delays.

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS10657134B2Selecting queries for execution on a stream of real-time data

Publication Date: 2020.05.19 AB INITIO TECHNOLOGY LLC

AI summary

A computer-implemented method for executing a query on data items located at different places in a stream of near real-time data to provide near-real time intermediate results for the query, as the query is being executed, the method including: from time to time, executing, by one or more computer systems, the query on two or more of the data items located at different places in the stream, with the two or more data items being accessed in near real-time with respect to each of the two or more data items; generating information indicative of results of executing the query; and as the query continues being executed, generating intermediate results of query execution by aggregating the results with prior results of executing the query on data items that previously appeared in the stream of near real-time data; and transmitting to one or more client devices the intermediate results of query execution, prior to completion of execution of the query.