Stream Computing Query Monitoring for Real-Time Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional database systems are not well-suited for real-time processing and analysis of large volumes of streaming data, leading to queries that do not reflect the most recent information due to the inability to efficiently store, index, and analyze in-flight data in streaming applications.

Innovation Solution

A stream computing application that monitors queries to identify data of interest, generates meta-query commands, and adjusts processing priorities and parameters to enhance the processing of in-flight data, ensuring it is more current, accurate, and complete by integrating it into a relational database more quickly.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional database systems are used to store and query data, then data storage and indexing are achieved, but real-time processing and analysis of streaming data is not possible

Engineering Contradiction:

Improvereal-time processing capabilityVSAvoidquery accuracy with recent information

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system segments data into two distinct pathways: historical data stored in the relational database and in-flight streaming data processed through the stream computing application. This segmentation allows each system to operate optimally - the database for structured storage and the streaming application for real-time processing - while together they provide both real-time processing capability and query accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent merges the relational database system with the stream computing application into an integrated architecture. The database provides storage and querying capabilities while the stream computing application provides real-time processing. The query monitor bridges these two systems, allowing queries to access both historical database data and current streaming data, thus achieving both real-time processing and query accuracy.

Inventive Principle:

Principle #5Merging (Combining)

2Loss of information

If all streamed data is processed and integrated into the database, then data completeness is improved, but processing overhead and time consumption increase significantly

Engineering Contradiction:

Improvedata completenessVSAvoiddata processing time

Core Design Contradiction:

Loss of informationVSLoss of time

Solution Approach 1:

Instead of processing all streamed data uniformly, the system applies partial action by selectively processing only those data tuples that match active query criteria. The query monitor identifies relevant queries and the stream computing application processes only the subset of streaming data that is currently of interest, rather than processing everything, thus reducing processing time while maintaining data completeness for relevant information.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system implements feedback through the query monitor, which continuously monitors active queries and uses this information to dynamically adjust processing priorities in the stream computing application. When queries are detected, the system feeds this information back to prioritize processing of relevant data tuples, ensuring that data completeness is maintained for queried information while minimizing processing time for non-queried data.

Inventive Principle:

Principle #23Feedback

3Productivity

If query monitoring and meta-query command generation are implemented, then processing priority adjustment is achieved, but system complexity increases

Engineering Contradiction:

Improvedata processing efficiencyVSAvoidsystem architecture complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The stream computing application is designed with multi-functionality, serving both as a real-time processing engine and as a query execution platform. Rather than requiring separate systems for streaming processing and query management, the same application performs both functions, reducing overall system complexity while maintaining processing efficiency through its dual role.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The query monitor acts as an intermediary component that bridges the relational database and the stream computing application. It monitors queries, generates meta-query commands, and communicates with the stream computing application to adjust processing priorities. This intermediary approach simplifies the architecture by providing a single coordination point rather than requiring complex direct integration between multiple systems.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11481253B2Managing the processing of streamed data in a data streaming application using query information from a relational database

Publication Date: 2022.10.25 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Queries are monitored in a database which receives input from a stream computing application to identify data of interest. Parameters defining the data of interest, which are preferably expressed as a logical query, are sent to the stream computing application, which then processes the in-flight streamed data satisfying the parameters in some special manner. In some embodiments, the stream computing application increases the processing priority of in-flight data satisfying the parameters. In some embodiments, the stream computing application applies additional processing steps to the in-flight data satisfying the parameters to provide enhanced data or metadata.