Stream Aggregation Current Function Retention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database systems are inadequate for processing continuous data streams as they are designed for finite data sets, lacking the flexibility needed for modern applications that generate unbounded event streams, such as sensor data and financial tickers, which require new techniques for managing and querying temporal data.
Innovation Solution
A method and system that process continuous data streams using streaming analytics, where attributes from current events or groups are retained during stream aggregation through summarization, grouping, and preservation using aggregate functions, group by functions, and a special current function in continuous query language, allowing for the output of relevant attributes alongside aggregated results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional database systems are used to process continuous data streams, then data can be stored and queried using standard SQL, but the system lacks the flexibility and temporal focus needed for unbounded event streams
Solution Approach 1:
The patent introduces dynamic query processing capabilities that allow queries to continuously adapt to incoming event streams. The system transitions from static batch processing to dynamic streaming processing, where queries automatically update as new events arrive, enabling the system to handle unbounded data streams with temporal awareness without requiring complete architectural redesign
Solution Approach 2:
The patent creates a unified query processing engine that can handle both traditional finite database queries and continuous stream queries using a single system. The query processor is designed to be universal, supporting both batch and streaming modes, thereby providing adaptability for continuous data processing while avoiding the complexity of maintaining separate specialized systems
2Productivity
If aggregate functions are applied to continuous event streams, then summarized results can be generated, but attributes from current events are lost during aggregation
Solution Approach 1:
The patent segments the query processing into distinct functional components: aggregation operators that compute summarized results and projection operators that preserve selected attributes. This segmentation allows the system to simultaneously perform aggregation for productivity while projecting specific attributes to prevent information loss, with each operator type handling its specialized function independently
Solution Approach 2:
The patent introduces an intermediary projection operator between the aggregation process and the final output. This intermediary component receives aggregated results and current event data, selectively preserves attributes that should not be lost during aggregation, and combines them with aggregation results to produce the final output, thereby mediating between the conflicting requirements of processing efficiency and information preservation
3Measurement precision
If group by function is used to collect events with identical values, then results can be grouped by specific attributes, but other attributes cannot be preserved in the output
Solution Approach 1:
The patent separates the grouping operation from the attribute selection operation. The group by operator precisely groups events by specified attributes to ensure measurement accuracy, while a subsequent projection operator independently selects which additional attributes to preserve in the output. This segmentation allows precise grouping while preventing loss of relevant non-grouped attributes through the projection stage
Data Source
AI summary
Techniques for processing event streams are provided that retain attributes or fields from a current event or group during stream aggregation. An exemplary technique includes receiving a stream of events having an input shape having a plurality of attributes and executing a query on the stream of events. The executing includes: (i) summarizing data of a first attribute of the plurality of attributes based on an aggregate function to generate values of data for a result-set, (ii) collecting data of events with values that are identical for a second attribute of the plurality of attributes based on a group by function to group the result-set by the second attribute, and (iii) preserving data of a third attribute of the plurality of attributes based on a current function. The technique further includes outputting events having an output shape having the first attribute, the second attribute, and the third attribute.