Data Stream Load Shedding via Quality-Based Tuple Deletion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data stream management systems face challenges in handling bursty data rates, leading to overloads that result in incomplete and inaccurate processing results due to insufficient resources, particularly in peak load situations where critical resources like computational power and memory are overwhelmed.
Innovation Solution
A method is introduced to analyze tuples in a data stream based on quality information, prioritizing the deletion of lower-quality tuples to shed load while retaining higher-quality ones, using data quality dimensions such as timeliness, completeness, and confidence to determine which tuples to keep or discard, thereby improving the overall quality of processing results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If load shedding strategies are used to delete excess data during overloads, then system resources are protected and processing can continue, but data loss occurs leading to incomplete and inaccurate results
Solution Approach 1:
The patent applies local quality by differentiating between high-quality and low-quality tuples within the data stream. Instead of uniform load shedding, the system identifies and prioritizes deletion of low-quality tuples (those with lower confidence, completeness, or timeliness scores) while preserving high-quality tuples. This selective approach maintains processing continuity during overloads while minimizing information loss, as the deleted tuples are those least likely to contribute to accurate results.
2Measurement precision
If resources are increased to handle peak load, then accurate processing results can be maintained, but system cost and complexity increase
Solution Approach 1:
The patent changes the quality parameters of tuples by calculating composite quality scores based on multiple dimensions (confidence, completeness, timeliness). During peak loads, the system dynamically adjusts which tuples are retained or deleted based on these quality parameter changes. This allows the system to maintain processing accuracy during overloads by preserving high-quality tuples without requiring proportional increases in computational resources, memory capacity, or bandwidth.
3Loss of information
If all data tuples are processed during bursty data rates, then complete results are achieved, but system overload occurs and processing quality degrades
Solution Approach 1:
The patent applies partial action by selectively processing only the most valuable tuples during bursty data rates. Instead of attempting to process all incoming tuples (which would cause overload and degrade quality), the system uses quality thresholds and prioritization to process a subset of high-quality tuples. This partial processing approach maintains acceptable data completeness in results while preventing system overload and preserving processing throughput during peak loads.
Data Source
AI summary
A system and method to delete overload in a data stream are described.A method of an embodiment of the invention may analyze data quality information in a data stream and delete data items that are found to be of lower than a desired data quality.In one embodiment, data items may be evaluated according to maximize a particular aspect of the utility of the data in a data stream.In one embodiment, a system of an embodiment of the invention may evaluate data quality in a data stream to suggest one or more actions to be performed to improve the data quality in the data stream. Further, the system of the embodiment of the invention may evaluate each suggested action to determine how the suggested action may impact the data quality in the data stream if performed.