Stream Join Model Threshold Filtering for Data Overload
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data stream processing systems face overload and memory issues when handling large quantities of data from multiple sources, often resulting in lag or cache overflow due to limited processing capabilities and delays in data transmission.
Innovation Solution
A data stream processing system that filters data streams by identifying and omitting dominated tuples within a window, using threshold functions like step, linear, or convex functions to generate filtered data streams, which are then applied to a stream join model, reducing processing resource usage and improving memory capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional systems process all incoming data from multiple sources, then data processing completeness is maintained, but system processing capacity becomes overloaded and memory is exhausted
Solution Approach 1:
The patent extracts and removes dominated tuples (redundant data) from the input stream before processing. By identifying and eliminating tuples that are dominated by other tuples based on threshold function evaluation, the system reduces the volume of data requiring full processing while maintaining the integrity of critical information, thus resolving the contradiction between processing completeness and system capacity
Solution Approach 2:
The patent changes the parameter of data representation by transforming tuples into a filtered format that removes redundancy. By applying threshold functions and dominance relationships, the system transforms the original data stream into a condensed representation that preserves essential information while reducing processing requirements
2Reliability
If the system caches data from multiple data streams to handle transmission delays, then data availability is improved, but processing capacity becomes overloaded due to excessive caching
Solution Approach 1:
The patent extracts only the necessary data for caching by removing dominated tuples before the caching operation. This ensures that the cache stores only non-redundant, essential data required for handling transmission delays, thereby improving data availability while preventing cache overload
Solution Approach 2:
The patent applies partial action by caching only the subset of data that is truly necessary (non-dominated tuples) rather than caching all incoming data. This selective approach provides sufficient data availability for handling delays without the excessive resource consumption that would result from caching redundant information
3Reliability
If the system processes large quantities of data from multiple sources, then data analysis completeness is maintained, but processing time increases causing lag
Solution Approach 1:
The patent performs preliminary filtering of dominated tuples before the main processing operation. By pre-processing the data stream to remove redundant information, the system reduces the workload for subsequent analysis operations, thereby maintaining analytical completeness while significantly reducing processing time and lag
Solution Approach 2:
The patent extracts and removes dominated tuples in advance of the main processing pipeline. This early elimination of redundant data ensures that subsequent analysis operations work only with essential information, preserving data analysis completeness while minimizing processing time losses
Data Source
AI summary
This application describes a data stream processing system for receiving and processing multiple data streams using a stream join model and based on the stream join model having a threshold function for which an output of the stream join model crosses the threshold two or less times (e.g., where the threshold function is a convex function, linear function, monotonic function, or other function having a similar property). The data stream processing system may generate filtered data streams using a number of techniques and algorithms without risk of false negatives and mis2169sing instances where an output of a stream join exceeds or violates a threshold condition. The data stream processing system can significantly reduce processing expense, particularly in cases where one or more devices have limited memory and where caching tuples from incoming data streams consumes significant processing resources.