Streaming Data Querying Without Persistence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data querying techniques require persistently storing streaming data, leading to high storage needs and low input/output efficiency, especially when processing multiple queries simultaneously, and are limited to backward-looking queries.
Innovation Solution
A system and method for querying streaming data in real-time without persistence, allowing queries to be executed against an infinite set of data arriving after the query submission, enabling forward-looking queries and simultaneous processing of multiple queries with improved I/O efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If streaming data is persistently stored for querying, then query capability is enabled, but storage space requirements increase significantly
Solution Approach 1:
The patent extracts only the necessary portion of streaming data for querying by implementing a time-window based filtering mechanism. Data records outside the specified time window are automatically discarded, while records within the window are retained for query processing. This selective extraction enables query capability on streaming data without requiring persistent storage of all historical data, thus resolving the contradiction between query reliability and storage volume.
2Adaptability or versatility
If multiple queries are processed simultaneously using existing techniques, then query coverage is comprehensive, but input/output efficiency decreases
Solution Approach 1:
The patent merges multiple query processing operations into a unified execution framework. When multiple queries are received, the system combines their filtering conditions and executes them simultaneously against the same streaming data flow. This consolidation reduces redundant I/O operations and data processing overhead, maintaining comprehensive query coverage while significantly improving input/output efficiency compared to processing each query independently.
3Loss of information
If backward-looking queries are used on streaming data, then historical event analysis is enabled, but real-time responsiveness is reduced
Solution Approach 1:
The patent implements dynamic query time-window adjustment that adapts to different analytical needs. The system allows the query time window to be dynamically configured and modified during runtime, enabling flexible switching between historical analysis (larger windows) and real-time responsiveness (smaller windows). This dynamic approach resolves the contradiction by allowing the same system to optimize for either historical event analysis or real-time responsiveness depending on the specific operational context.
Data Source
AI summary
The present teaching relates to streaming data querying and providing query results. In one example, a query is received for events happening after receipt of the query. A plurality of data records are obtained, where each of the data records corresponds to an event happening after receipt of the query. For each of the plurality of data records, it is determined whether the data record matches the query. Based on the determining, one or more query results are determined and provided as a response to the query.