Dynamic Maximum Delay Adjustment for Time Series Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in effectively handling late or delayed time series data points, which can lead to incomplete data sets and imbalanced timeliness and completeness in real-time analytics, especially when dealing with network latency and varying delay times.
Innovation Solution
A data quantizer system dynamically adjusts maximum delay values based on statistical evaluations, using weighted moving averages and variance to determine the inclusion and expiration of late data points, ensuring timely and complete data publication to streaming analytics engines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If a fixed maximum delay value is used for data publication, then data timeliness is maintained, but data completeness deteriorates when network latency causes delays
Solution Approach 1:
The patent applies dynamics by transitioning from a fixed maximum delay value to a dynamically adjusted maximum delay value. The system continuously monitors actual data delays and adjusts the maximum delay threshold accordingly, allowing it to adapt to varying network conditions and delay patterns, thus resolving the contradiction between maintaining timeliness and ensuring completeness
Solution Approach 2:
The patent implements feedback by using statistical evaluation of actual data delays to adjust the maximum delay value. The system monitors delay patterns, analyzes them statistically, and uses this feedback to dynamically modify the maximum delay threshold, creating a closed-loop control mechanism that balances timeliness and completeness
2Loss of time
If late data points are excluded from processing, then data timeliness is improved, but data completeness deteriorates
Solution Approach 1:
The patent applies parameter changes by modifying the maximum delay parameter based on statistical evaluation of actual delay patterns. Instead of using a static threshold, the system adjusts this parameter dynamically, allowing the acceptance criteria for late data points to change over time based on observed network conditions and delay characteristics
3Device complexity
If a static maximum delay value is used, then system simplicity is maintained, but adaptability to varying network conditions deteriorates
Solution Approach 1:
The patent transitions from a static to a dynamic maximum delay value, enabling the system to adapt to varying network conditions. This dynamic adjustment mechanism provides versatility in handling different delay scenarios while maintaining a relatively simple implementation through automated statistical evaluation and adjustment
Data Source
AI summary
Described are systems, methods, and techniques for collecting, analyzing, processing, and storing time series data and for evaluating and determining whether and how to include late or delayed data points for inclusion when publishing or storing the time series data. Maximum delay values can identify a duration for waiting for late or delayed data, such as prior to publication. In some examples, maximum delay values can be dynamically adjustable based on a statistical evaluation process. For late or delayed data points that are received after the maximum delay elapses, some data points can be included in the stored time series data, such as if they are received in the same order that they are generated.