Data Stream Quality Aggregation via Jumping Windows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data stream technologies face challenges in managing data quality, particularly due to sensor limitations and environmental factors, which can lead to incorrect decisions and increased costs, and there is a need for efficient storage and processing of large volumes of sensor data with associated quality information.
Innovation Solution
A system comprising a data stream engine with a data manager, data quality manager, and aggregation manager that determines and aggregates data quality values, allowing for the inclusion of data quality information in data streams and efficient storage in relational databases using jumping windows to reduce resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data quality information is included for each sensor data item, then data quality management is improved, but storage and computing overhead increases significantly
Solution Approach 1:
The patent combines multiple data quality metrics (accuracy, completeness, timeliness, consistency) into a single aggregated data quality value for each time window. This merging approach allows comprehensive data quality assessment while reducing the quantity of stored information, directly resolving the contradiction between improved data quality management and reduced storage overhead.
Solution Approach 2:
The patent segments the continuous data stream into discrete time windows, calculating aggregated data quality values for each window rather than for every individual data point. This segmentation reduces the total number of quality assessments needed, lowering computing and storage overhead while maintaining effective data quality management at the window level.
2Quantity of substance
If aggregated data quality values are calculated for time windows, then storage overhead is reduced, but data quality management granularity is decreased
Solution Approach 1:
The patent implements a dynamic time window mechanism where the window size and sliding interval can be adjusted based on data characteristics and quality requirements. This dynamic approach allows the system to adapt granularity levels - using smaller windows for fine-grained quality assessment when needed and larger windows for reduced overhead, thus resolving the contradiction between storage efficiency and management granularity.
Data Source
AI summary
A method and system are described for including data quality in data streams. An example method may include obtaining a first group of data items, each data item including one or more data attribute values. A first group of data quality items may be determined, each data quality item including one or more data quality attribute values associated with one of the data items of the first group. A first aggregated data quality value may be determined based on the first group of data quality items. A first data stream interval including the first group of data items and the first aggregated data quality value may be output.