Distributed Data Flow Rerouting for Message-Driven Fault Mitigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed systems, particularly those that are event/message-driven, identifying and mitigating faults quickly is challenging due to the lack of local response information, making traditional RPC-based fault mitigation impractical, and the need for rapid fault mitigation in real-time applications is critical.
Innovation Solution
Implementing a system with observability instrumentation to collect measurements from service instances, using health monitoring to identify faulty entities, and rerouting data flows to alternative service instances through circuit breaking to maintain quality-of-service requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional RPC-based fault mitigation is used, then local response information is available for fault identification, but it becomes impractical in event/message-driven distributed systems where responses do not return to the caller node
Solution Approach 1:
The patent introduces an intermediary mechanism (observability instrumentation and health monitoring system) that mediates between the caller and the distributed system components. Instead of relying on direct local response information from RPC calls, the intermediary collects measurements and health status from multiple sources across the distributed system, enabling fault identification without requiring responses to return to the original caller node.
2Reliability
If data flows are distributed to multiple destination service instances, then system reliability is improved through redundancy, but fault identification becomes more difficult due to lack of local response information
Solution Approach 1:
The patent implements a feedback mechanism where observability instrumentation continuously collects measurements from service instances and health monitoring systems. This feedback loop provides real-time information about the status of distributed data flows and service instances, enabling automatic fault detection and rerouting decisions without requiring complex manual analysis of multiple distributed components.
Solution Approach 2:
The patent segments the fault detection and mitigation functionality into independent components: observability instrumentation for data collection, health monitoring for analysis, and rerouting logic for action. This segmentation allows each component to focus on a specific aspect of fault mitigation, making the overall system more manageable and effective in identifying faults across multiple destination service instances.
3Productivity
If quick fault identification is implemented in distributed systems, then system performance is maintained through rapid mitigation, but the complexity of the monitoring and measurement system increases
Solution Approach 1:
The patent implements preliminary action by pre-deploying observability instrumentation within service instances and establishing health monitoring systems before faults occur. Measurements and health status are continuously collected and analyzed in advance, so when a fault occurs, the system can immediately identify and respond to it without needing to implement complex real-time analysis mechanisms at the moment of failure.
Data Source
AI summary
Embodiments include methods, electronic device, storage medium, and computer program for fault mitigation in a distributed system. In one embodiment, a method comprises obtaining measurements related to one or more one-way data flows that are from one or more source service instances and that are to be distributed to one of at least two destination service instances in the distributed system; determining the obtained measurements indicating that distribution of a one-way data flow within the one or more one-way data flows to a destination service instance of the at least two destination service instances fails to comply with a quality-of-service requirement; and causing reroute of the one-way data flow to be distributed to another destination service instance instead of the destination service instance.