Query Rewind Mechanism for Continuous Data Stream Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing database systems face challenges in efficiently processing and querying large volumes of continuously collected data due to overhead in workflow scheduling, memory management, and inter-process communication, leading to suboptimal performance and latency in data access.

Innovation Solution

The implementation of a Super Iterative Continuous Query (SICQ) mechanism that processes data in chunk-by-chunk fashion using a query engine with a rewind mechanism, sustaining the query state across cycles and eliminating the need for frequent setup and teardown, allowing for real-time analysis and incremental computations with flexible granularities.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional database systems process continuous data streams using workflow scheduling, then data can be collected and stored, but processing latency and overhead increase significantly

Engineering Contradiction:

Improvedata processing throughputVSAvoidquery execution latency

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent implements continuous query processing where the query execution plan remains active and processes data chunks continuously without repeated setup and teardown. The query instance sustains its state across multiple data chunks, eliminating the need for frequent workflow scheduling overhead and reducing processing latency while maintaining high throughput

Inventive Principle:

Principle #20Continuity of useful action

2Ease of operation

If queries are executed repeatedly on each data chunk with setup and teardown, then data processing can be performed, but overhead in memory management and inter-process communication increases

Engineering Contradiction:

Improvequery execution simplicityVSAvoidmemory management overhead

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent merges multiple separate query executions into a single continuous query instance that processes multiple data chunks. By combining the query setup, execution, and state management into one sustained process, the system eliminates repeated memory allocation and inter-process communication overhead while maintaining operational simplicity through a unified query interface

Inventive Principle:

Principle #5Merging (Combining)

3Reliability

If the query instance is torn down and restarted for each data chunk, then resource cleanup occurs, but performance degradation occurs due to frequent setup and teardown

Engineering Contradiction:

Improveresource managementVSAvoidquery processing efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent performs preliminary resource allocation and query plan compilation when the continuous query is first created, then sustains this prepared state across multiple data chunk processing cycles. This preliminary action ensures resources are properly managed while avoiding repeated setup overhead, maintaining both reliability and high processing efficiency

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS8620945B2Query rewind mechanism for processing a continuous stream of data

Publication Date: 2013.12.31 MICRO FOCUS LLC

AI summary

A query engine receives a continuous stream of data and applies a Structured Query Language (SQL) query to the data. The data is processed on a chunk-by-chunk basis with a query rewind mechanism that sustains the SQL query as a single long-standing query that allows a state of the SQL query to be maintained.