Data-Continuous SQL Process for Stream Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database management systems face challenges in efficiently processing and querying large volumes of continuously incoming data due to complex queries and the need for timely data access, leading to performance issues.
Innovation Solution
The introduction of Data-Continuous Structured Query Language (SQL) Process (DCSP) using Relation Valued Functions (RVFs) that process continuous data streams, enabling automatic information derivation through continuous querying and computation, with RVFs acting as relational transformations triggered by stream inputs, timers, or event-conditions, and utilizing a station-hosted RVF and FIFO stream container for asynchronous communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional database management systems process large volumes of continuously incoming data with complex queries, then data access and manipulation capabilities are maintained, but performance deteriorates due to insufficient query completion time
Solution Approach 1:
The patent segments the continuous data stream processing into discrete micro-batches that can be processed independently and in parallel. This segmentation allows the database system to handle continuous data ingress by breaking it into manageable units, improving throughput while maintaining query responsiveness through parallel processing of these segments
Solution Approach 2:
The system performs preliminary actions by pre-processing and buffering incoming data streams before they reach the main query processing layer. This preliminary data preparation and staging allows complex queries to execute faster by working with pre-organized data, reducing query completion time while maintaining high data processing throughput
2Quantity of substance
If database systems store and manage increasingly large volumes of data, then data capacity and information availability are improved, but query execution efficiency deteriorates
Solution Approach 1:
The patent applies local quality by creating specialized data structures and indexing mechanisms tailored to specific query patterns and data types within the large data volume. This allows the system to optimize query execution for different portions of the data based on their access patterns, maintaining high query efficiency even as overall data capacity increases
Solution Approach 2:
The system introduces additional dimensions for data organization beyond traditional flat storage, implementing multi-dimensional indexing and hierarchical data structures. This dimensional approach allows efficient navigation and retrieval within large data volumes by providing multiple access paths, thus maintaining query execution efficiency as data capacity scales
3Reliability
If applications require continuous data collection and processing, then real-time information availability is improved, but system complexity and processing challenges increase
Solution Approach 1:
The patent merges the data collection, buffering, and initial processing functions into an integrated stream processing module that operates within the database system. This consolidation reduces system complexity by eliminating separate components while maintaining real-time data availability through the unified processing pipeline that handles continuous data ingress and query serving
Data Source
AI summary
One embodiment is a method that repeatedly triggers a User Defined Function (UDF) that generates a data stream. A data continuous Structured Query Language (SQL) query is applied to the data stream to construct a data continuous SQL process (DCSP) which executes in a database.