Pipe Worker Sequence Number Filtering for Distributed Execution Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems face challenges in achieving high reliability when executing plural distributed programs connected through a pipe, as inconsistencies in execution states can occur due to asynchronous storage times, leading to potential failures and reduced system reliability.
Innovation Solution
The proposed solution involves a distributed execution system with output-side and input-side pipe workers that manage data transmission and sequence numbers to ensure consistent execution states, where output data is only forwarded if the sequence number is greater than the restore sequence number, preventing re-transmission of earlier data and maintaining state consistency across program workers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If execution states are stored at different times for each program, then individual program recovery is possible, but inconsistency between replaced execution states occurs
Solution Approach 1:
The system performs preliminary actions by having all programs store their execution states at a unified checkpoint before allowing recovery. The checkpoint manager coordinates the timing of state storage across all programs, ensuring that when recovery occurs, all programs restore to consistent states that were captured at the same logical point in time, preventing inconsistency between replaced execution states
Solution Approach 2:
The system implements feedback mechanisms where the checkpoint manager monitors and coordinates state storage across all programs. When a program attempts to store its execution state, it receives feedback about the current checkpoint status and adjusts its storage timing accordingly. This feedback loop ensures that all programs store states in a coordinated manner, maintaining consistency while enabling individual recovery
2Reliability
If pipe workers forward all output data, then complete data transmission occurs, but re-transmission of already processed data causes inefficiency
Solution Approach 1:
The pipe worker performs preliminary action by checking the sequence number of incoming output data against the sequence number of the last forwarded data before actually forwarding. This preliminary check prevents re-transmission of already processed data, improving efficiency while maintaining complete transmission of all unique data through the pipe
Solution Approach 2:
The system replaces the mechanical approach of blindly forwarding all data with an intelligent filtering mechanism based on sequence number comparison. The pipe worker uses logical comparison instead of simple data forwarding, substituting a dumb transmission system with a smart, sequence-aware system that automatically filters out duplicates while preserving all unique data
Data Source
AI summary
A distributed execution system includes an output-side pipe worker that operates on a node same as an output-side worker realized by a first distributed program, and an input-side pipe worker that operates on a node same as an input-side worker realized by a second distributed program, receives output data on the output-side worker from the output-side pipe worker, and transfers it to the input-side worker, in which the output-side pipe worker acquires, from the output-side worker, output data together with a sequence number indicating an order of the output data to be transmitted to the input-side worker, acquires a restore sequence number corresponding to an execution state of the input-side worker, compares the sequence number and the restore sequence number, and does not forward, to the input-side pipe worker, the output data acquired together with the sequence number indicating the order equal to or earlier than the restore sequence number.