Standby Node Tick Buffer for High-Speed Data Stream Sync
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing high availability techniques for stream-based applications are inadequate for handling high data rates and real-time updates, often failing to continue data stream processing after a node failure, especially in systems like stock trading applications where data volatility and system crashes can lead to significant disruptions.
Innovation Solution
A method and system for synchronizing a standby node with a live node in stream-based applications, where both nodes process the data stream simultaneously, with the standby node temporarily storing ticks in a tick save buffer, identifying new symbols, requesting metadata, and synchronizing state with the live node, allowing seamless continuation of data stream processing upon live node failure without the need for checkpoints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transaction-based updates are used for high availability, then data consistency is improved, but processing speed deteriorates
Solution Approach 1:
The patent segments the high availability mechanism into two distinct phases: initial synchronization phase where transaction-based updates ensure data consistency, and normal operation phase where the standby node processes ticks independently. This segmentation allows the system to use transaction-based updates only when necessary (during initial synchronization) rather than continuously, thus maintaining data consistency while avoiding ongoing processing speed degradation.
2Measurement precision
If the standby node waits for full synchronization before processing, then data accuracy is improved, but system responsiveness deteriorates
Solution Approach 1:
The patent applies preliminary action by having the standby node perform initial synchronization with the live node before taking over processing. During this preliminary phase, the standby node receives and processes all ticks to build an accurate initial state. Once synchronized, the standby node can immediately take over without waiting, ensuring both data accuracy (through preliminary synchronization) and system responsiveness (through immediate takeover capability).
Solution Approach 2:
The patent uses a tick save buffer as an intermediary mechanism during the transition from live to standby node. When the standby node takes over, it uses this buffer to temporarily store ticks that arrive during the transition period, ensuring no data loss while maintaining continuous processing. This intermediary buffer allows the system to switch nodes without waiting for perfect synchronization timing, thus improving responsiveness while maintaining accuracy.
3Productivity
If the standby node processes ticks independently after synchronization, then processing speed is improved, but data consistency deteriorates
Solution Approach 1:
The patent implements a copying mechanism where the standby node creates a complete copy of the live node's state during initial synchronization, including all symbol metadata and tick histories. This copying ensures that the standby node starts with identical data to the live node. After synchronization, the standby node processes ticks independently using this copied state, achieving high processing speed while maintaining data consistency through the initial accurate copy and periodic state comparisons.
Data Source
AI summary
Embodiments of the present invention provide methods and systems for high availability in data-stream applications using separate but synchronized nodes. Initially, a live node is designated to provide the data stream. Upon startup or restart, a standby node synchronizes its state with the live node. The initial synchronization is performed in two phases. In the first phase, the standby node connects to the same data stream (or a data stream with same content) and obtains the same symbols as the live node. In the second phase, the standby node obtains metadata from the live node for various symbols, such as symbols that are new to the standby node. Once synchronized, the live and the standby nodes then continue with separately processing the data stream.