ISTREAM Operator Column Change Detection in CEP
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Complex Event Processing (CEP) systems, specifically the ISTREAM operator, output unnecessary data by considering all columns, failing to efficiently identify changes in only a subset of columns of interest, leading to non-deterministic and resource-intensive processing.
Innovation Solution
The proposed solution involves initializing a stream of data, setting a time interval, identifying and monitoring specific columns for changes, and outputting only the changed values, utilizing NOT IN semantics to filter out irrelevant data, thus providing deterministic and efficient processing of streaming data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the ISTREAM operator monitors all columns to ensure complete data accuracy, then data completeness is improved, but processing efficiency deteriorates due to unnecessary data output and increased resource consumption
Solution Approach 1:
The patent extracts and monitors only the specific subset of columns that have changed between time intervals, rather than processing all columns. This is achieved by identifying changed columns through a difference operation and then monitoring only those specific columns for subsequent changes, thereby eliminating unnecessary processing of unchanged columns while maintaining data accuracy for the relevant subset.
Solution Approach 2:
The patent segments the monitoring process into distinct phases: first identifying which columns have changed using a difference operation, then separately monitoring only those changed columns in subsequent operations. This segmentation allows the system to focus computational resources on the relevant data subset, improving processing efficiency while maintaining reliability for the columns of interest.
2Loss of information
If the ISTREAM operator processes all columns to ensure no information is lost, then information completeness is improved, but resource consumption deteriorates due to handling unnecessary data
Solution Approach 1:
The patent extracts only the changed columns from the full relation and monitors subsequent changes only in those extracted columns. This extraction approach prevents information loss for the relevant columns while avoiding the resource consumption associated with processing and transmitting unchanged columns, thus resolving the contradiction between information completeness and resource usage.
3Reliability
If the ISTREAM operator outputs data for all columns to maintain data integrity, then data integrity is improved, but output efficiency deteriorates due to inclusion of unnecessary data
Solution Approach 1:
The patent extracts and outputs only the changed columns rather than all columns. By identifying which columns have changed through a difference operation and then monitoring and outputting only those specific columns, the system maintains data integrity for the relevant information while significantly improving output efficiency by eliminating unnecessary data transmission.
Data Source
AI summary
One embodiment of the invention includes a method of processing streaming data. The method includes initializing a stream of data and setting a time interval to apply to the stream of data. The time interval comprises a window for analyzing the data within the stream of data. The method further includes identifying one or more columns within the stream of data, designating one or more of the columns to be monitored for differences within the data over the time interval, and monitoring the designated columns over the time interval. Further, the method includes determining that at least one value from at least one of the designated columns has changed and in response to at least one value changing, outputting the changed values from the designated columns.