Projecting In-Flight Streamed Data Changes in Relational Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems are not well-suited for real-time processing and analysis of large volumes of streaming data, leading to obsolete query results and management decisions based on stale information, as they do not account for future changes and in-flight data in streaming applications.
Innovation Solution
Profiling data from a streaming application is used to project changes in a relational database by constructing operator graph profile data, which is then used to predict the numbers, types, and attributes of tuples that will be produced, allowing for more accurate and complete information to be provided to users and improving database management efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional database systems are used to store and query data, then data storage and retrieval are supported, but real-time processing and analysis of streaming data cannot be performed efficiently
Solution Approach 1:
The system performs preliminary actions by collecting trace data during execution instances to construct operator graph profile data before queries are executed. This profile data characterizes the streaming application's behavior and is used to project future database changes, enabling the database system to anticipate and prepare for upcoming data changes rather than reacting to them after they occur.
Solution Approach 2:
The system dynamically adapts to streaming data characteristics by using profile data that captures the actual behavior of the streaming application during execution. The operator graph profile data is constructed from trace data collected during real execution instances, allowing the system to dynamically adjust its projections based on actual observed patterns rather than static assumptions.
2Ease of operation
If database systems separate data storage from data access and manipulation, then data management is simplified, but the system cannot account for future changes and in-flight data in streaming applications
Solution Approach 1:
The system introduces an intermediary layer between data storage and data access by using operator graph profile data as a mediator. This profile data acts as an intermediate representation that captures the relationship between streaming application execution and future database changes, allowing the system to bridge the gap between current stored data and future data states without complicating the fundamental separation of storage and access operations.
3Productivity
If conventional query techniques are used, then simple data retrieval is achieved, but query results become obsolete or misleading when large volumes of streaming data are processed
Solution Approach 1:
The system implements feedback by using trace data collected during execution instances to construct operator graph profile data, which then feeds back into the query process to project future database changes. This feedback loop allows the system to continuously improve its understanding of streaming application behavior and adjust its projections accordingly, ensuring query results remain accurate even when processing large volumes of streaming data.
Data Source
AI summary
Profiling data characterizing a data streaming application is used to project changes to a relational database resulting from current in-flight streamed data. Preferably, the streaming application produces tuples which are entered into the relational database. Trace data is collected during previous execution of the streaming application to construct operator graph profile data showing likely paths of tuples through multiple processing elements of the streaming application. Responsive to a query, agent(s) residing within the computer system(s) supporting the streaming application query in-flight tuples in one or more buffers of the streaming application. The responses to the agent queries are analyzed using the operator graph profile data to project tuples which will be output to the database. Projected changes to the database may alternatively be used for other purposes, e.g., creating database metadata structures; reorganizing data inserts; regulating query governors; and/or updating database statistics.