Stream Data Processing Device Timestamp Order Maintenance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stream data processing systems face challenges in maintaining the correct order of time-stamped tuples due to out-of-order arrivals from multiple data sources, leading to inconsistencies in processing.
Innovation Solution
A method involving a processing holdback period is implemented, where system time stamps are attached to incoming tuples, and data source-based queues are sorted and managed using a chronological alignment tree to ensure tuples are processed in the correct order of their data time stamps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tuples are processed in parallel from multiple data sources, then processing speed is improved, but the order of time stamps cannot be maintained in out-of-order cases
Solution Approach 1:
The patent applies preliminary action by attaching system time stamps to tuples at the point of reception (before processing) and pre-sorting tuples based on these time stamps. This preliminary organization of data in chronological order enables subsequent parallel processing to maintain timestamp order without requiring complex coordination during processing, thus resolving the contradiction between processing speed and timestamp order maintenance.
2Adaptability or versatility
If damping function is used to weight tuples by time stamp, then recent data is emphasized, but correct timestamp order cannot be maintained in out-of-order cases
Solution Approach 1:
The patent performs preliminary sorting of tuples based on system time stamps before processing. By organizing tuples in correct chronological order in advance, the system can then apply damping functions for data weighting without compromising timestamp order maintenance, thus resolving the contradiction between adaptability and reliability.
3Ease of operation
If independent work areas are created for each data source, then partial sorting of tuples is achieved, but overall tuple order remains indeterminate
Solution Approach 1:
The patent merges the independently processed tuples from multiple data sources by sorting them based on system time stamps after processing. This combining step ensures that while each data source can be processed independently (maintaining ease of operation), the final output maintains overall timestamp order (ensuring reliability).
4Productivity
If tuples from multiple data sources are combined before processing, then parallel processing is enabled, but timestamp order inconsistencies occur due to path delays
Solution Approach 1:
The patent attaches system time stamps to tuples at reception and performs preliminary sorting before parallel processing. This preliminary action ensures that even when tuples from multiple data sources are combined and processed in parallel, the original timestamp order is preserved, thus resolving the contradiction between parallel processing capability and timestamp order accuracy.
Data Source
AI summary
An input tuple including a data time stamp is assigned by a data source; a pending tuple in which a system time stamp at the time the input tuple was received is assigned to the input tuple; and the pending tuple is stored in a queue, one queue for each data source. The queues are sorted in the order of the data time stamps of the pending tuples at the head of each queue; and if, in the queue at the head of which is stored the pending tuple having the smallest data time stamp value, the value of the current system time stamp is greater than the value of a processing pending period added to the system time stamp of the pending tuple at the head of the queue, then the input tuple is acquired from the pending tuple at the head of the queue for an input stream.