Phantom Query System for Data Stream Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data stream management systems, such as Gigascope, inefficiently handle multiple aggregation queries by not sharing computation, leading to unnecessary work and increased costs due to incorrect identification of phantoms.
Innovation Solution
Implementing a phantom query system that groups packets using attributes other than those used for user queries, allowing shared computation between multiple aggregation queries, thereby reducing processing costs by maintaining a hash table for common attributes and using phantom queries to derive results for user queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple aggregation queries are evaluated independently without shared computation, then each query can be processed individually, but processing cost and time increase significantly
Solution Approach 1:
The patent merges the evaluation of multiple aggregation queries by identifying and maintaining common sub-expressions (phantoms) that are shared across different queries. Instead of evaluating each query independently, the system combines their evaluation by maintaining a single hash table for common attributes that serves multiple queries simultaneously, thereby reducing redundant computation and improving productivity while lowering processing costs.
Solution Approach 2:
The patent creates phantom aggregations that serve multiple user queries simultaneously. A single phantom aggregation on common attributes can be used to answer multiple different user queries by combining its results with additional attribute-specific computations. This multi-functionality allows the system to serve multiple queries with a single computational foundation, improving efficiency and reducing processing costs.
2Productivity
If common sub-expression analysis is used to share computation between queries, then processing efficiency improves, but incorrect identification of phantoms leads to additional work with no benefit
Solution Approach 1:
The patent applies local quality by making phantom identification query-specific rather than universal. Each query's common sub-expressions are identified in the context of that specific query's needs, allowing the system to maintain phantoms that are locally optimized for each query rather than attempting a one-size-fits-all approach. This reduces the complexity of phantom identification while maintaining high computation sharing efficiency.
3Ease of manufacture
If Gigascope evaluates multiple aggregation queries independently with no shared computation, then implementation is simple, but processing cost increases and efficiency decreases
Solution Approach 1:
The patent segments the query evaluation process into two distinct phases: (1) identification and maintenance of phantom aggregations for common sub-expressions, and (2) evaluation of user queries using these phantoms. This segmentation allows the system to maintain simplicity in the underlying architecture while introducing optimization layers that improve productivity. The phantom maintenance layer operates independently from the query evaluation layer, preserving implementation simplicity while enabling efficient computation sharing.
Data Source
AI summary
A system for a data stream management system includes a filter transport aggregate for a high speed input data stream with a plurality of packets each packet comprising attributes. The system includes an evaluation system to evaluate the high speed input data stream and partitions the packets into groups the attributes and a table, wherein the table stores the attributes of each packets using a hash function. A phantom query is used to define partitioned groups of packets using attributes other than those used to group the packets for solving user queries without performing the user queries on the high speed input data stream.