Distributed Data Stream Processing for Fault Tolerance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data stream processing systems in multi-pipelined environments often fail to maintain data integrity during errors, particularly in applications like air traffic control, where data loss can be hazardous due to their focus on end states rather than in-transit data status.
Innovation Solution
A distributed architecture with a master processor creating data packets encapsulating transaction summaries and a slave processor emulating the master's transactional state to regenerate the data stream, ensuring continuous data flow even in case of errors, utilizing a communications network for data streams and processors that can be remotely located for increased fault tolerance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional transaction processing techniques are used to preserve data integrity, then data can be restored to its last known valid state, but the flow of data in transit is interrupted and data loss occurs
Solution Approach 1:
The system performs preliminary actions by creating checkpoints and transaction summaries before errors occur. The master processor generates transaction summaries that capture the state of data processing at various points, allowing the slave processor to resume from the last valid checkpoint without losing in-transit data. This preliminary preparation enables continuous data flow recovery without interruption.
Solution Approach 2:
The invention uses copying by maintaining a slave processor that replicates the master processor's state and continues processing. The slave processor receives and processes copies of transaction summaries, creating a redundant processing path that can take over seamlessly when the master fails, thereby preventing data loss and maintaining continuous data flow.
2Ease of operation
If a master processor handles all data stream processing, then centralized control is achieved, but single point of failure risk increases
Solution Approach 1:
The invention extracts the critical processing functions from the master processor and assigns them to a slave processor. The slave processor is given the capability to independently process transaction summaries and regenerate data streams, removing the single point of failure. This extraction maintains centralized control architecture while distributing the actual processing burden and failure risk.
Solution Approach 2:
The system changes the operational parameters of the slave processor from a passive standby state to an active processing state capable of autonomous operation. By modifying the slave's parameters to include full processing capability and transaction summary handling, the system achieves both centralized control and high fault tolerance, as the slave can immediately take over when the master fails.
3Loss of energy
If the slave processor remains idle during normal operation, then resource consumption is reduced, but response time during failure increases
Solution Approach 1:
The slave processor performs preliminary actions by continuously receiving and processing transaction summaries from the master processor even during normal operation. This keeps the slave synchronized with the current data state and ready for immediate failover. The preliminary processing of transaction summaries ensures that when failure occurs, the slave can resume processing without delay, minimizing both energy waste and failover time.
Solution Approach 2:
The invention maintains continuity of useful action by having the slave processor continuously process transaction summaries and maintain synchronization with the master. Rather than remaining completely idle, the slave performs useful preparatory work that reduces failover time. This continuous action ensures that the slave is always ready to take over, balancing resource consumption with rapid response capability.
Data Source
AI summary
A distributed architecture and method for maintaining the integrity of data streams within a multi-pipelined processing environment. The architecture comprising a communications network for carrying a plurality of data streams and a master processor adapted to process one or more messages in at least one of the data streams, the message processing including the creation of one or more data packets within the stream, each packet encapsulating at least a transaction summary of the data that has been processed. The architecture further comprising at least one slave processor per master processor adapted to emulate the transactional state of the master processor by regenerating the data stream as a result of processing the one or more data packets, whereupon in response to an error event on the master processor, the slave processor acts to avoid interrupting the data stream by generating one or more successive data packet(s). Hence, the architecture and method serve as a high availability, robust fault tolerant system, mitigating against the loss of data within data streams.