Data Forwarder Multi-Pipeline Recovery Mechanism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data forwarding systems face challenges in reliably transmitting transactions due to system instabilities, such as crashes, where real-time data may be lost, and inefficiencies arise from the need to read data from disk or non-volatile storage, leading to dropped transactions and increased costs.
Innovation Solution
Implementing multiple data pipelines and queues, including live and stale data paths, where transactions are stored in non-volatile storage and can be sent through either in-memory or stale data paths to ensure resilience and flexibility, allowing for the prioritization of live data while compensating for system instabilities by using stale data paths when necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in volatile memory for real-time forwarding, then forwarding speed is improved, but data loss occurs during system crashes
Solution Approach 1:
The patent applies preliminary action by writing data to non-volatile storage (disk) before it is needed for forwarding. This ensures that even if the system crashes, the data remains persisted and can be recovered. The data forwarder maintains both in-memory data for fast forwarding and disk-based storage for persistence, with a recovery mechanism that replays transactions from disk if data loss occurs.
2Reliability
If data is read from disk for every transaction, then data persistence is ensured, but forwarding efficiency decreases
Solution Approach 1:
The patent segments the data forwarding system into two distinct paths: a fast path for live data that operates entirely in memory without disk access, and a slow path for stale data that reads from disk. This segmentation allows the system to achieve both high throughput for normal operations and data persistence, by only accessing disk when necessary for recovery or when data cannot be forwarded through the fast path.
3Reliability
If multiple data paths are implemented for resilience, then system robustness is improved, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary component called the transaction log that mediates between the fast in-memory data path and the slow disk-based path. The transaction log acts as a buffer and recovery mechanism, allowing the system to maintain simple fast-path operations while providing robustness through the intermediary log that can supply data if the fast path fails.
4Loss of time
If in-memory data paths are used for live data, then forwarding latency is reduced, but data loss risk increases during system instability
Solution Approach 1:
The patent applies beforehand cushioning by maintaining a cushion of data in the transaction log on non-volatile storage. This cushion protects against data loss by ensuring that even if in-memory data is lost during a crash, the data can be recovered from the transaction log cushion that was prepared in advance.
Data Source
AI summary
In accordance with implementations of the present disclosure, a backup of live data received by a data forwarder is generated at the data forwarder while the live data is provided to a real-time data pipeline for forwarding from the data forwarder. A first portion of the live data is recovered from the backup to a stale data pipeline of the data forwarder. A request to forward the live data to a destination node is received by the data forwarder. In response to the request data is forwarded to the destination node, where the first portion of the live data from the stale data pipeline is added to a second portion of the live data from the real-time data pipeline in the response based on determining headroom remains to reach an amount of the data identified to include in the response.