Distributed System Partition Management via Heartbeat Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face performance degradation due to the lack of automatic reconfiguration and node failure handling, especially in multi-subnet environments, where conventional consensus algorithms like RAFT cannot adapt well and lead to unnecessary node replacement and data migration.
Innovation Solution
A method involving heartbeat messages to determine node states and partition statuses, allowing for decentralized node discovery and self-healing, which avoids unnecessary node replacement and improves system performance by differentiating between node malfunctions and network isolation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional consensus algorithms like RAFT are used to ensure data consensus in distributed storage systems, then system reliability is improved, but system performance deteriorates due to unnecessary node replacement and data migration
Solution Approach 1:
The system divides nodes into different partitions based on their network locations (subnets). By segmenting the node management approach, the system can independently handle failures in different partitions without triggering system-wide node replacement and data migration, thus maintaining reliability while improving performance.
Solution Approach 2:
The system changes the parameter of node state detection by introducing partition-aware heartbeat mechanisms. Instead of treating all node failures uniformly, the system modifies the failure detection parameter to consider partition context, enabling differentiated responses that prevent unnecessary performance degradation.
2Device complexity
If conventional consensus algorithms are used without automatic reconfiguration capability, then implementation simplicity is maintained, but adaptability to multi-subnet environments deteriorates
Solution Approach 1:
The system introduces dynamic partition detection and automatic reconfiguration capabilities that adapt to multi-subnet environments. The partition information is dynamically obtained and used to adjust node management strategies, enabling the system to adapt to different network topologies without fundamentally changing the consensus algorithm structure.
Solution Approach 2:
The system implements self-service through automatic reconfiguration when node failures occur. The master node automatically detects failures, determines affected partitions, and reconfigures the system without manual intervention, maintaining implementation simplicity while improving adaptability to multi-subnet environments.
3Measurement precision
If heartbeat messages are sent to all slave nodes without partition consideration, then node monitoring completeness is improved, but network overhead increases
Solution Approach 1:
The system segments the slave nodes into different partitions based on network subnets. Heartbeat messages are then selectively sent only to slave nodes within the same partition as the master node, reducing network overhead while maintaining complete monitoring within each partition through partition-aware routing.
Data Source
AI summary
Embodiments of the present disclosure relate to a method, a device and a computer program product for managing a distributed system. The method comprises sending heartbeat messages from a master node to a plurality of slave nodes, the master node and the plurality of slave nodes being included in a plurality of nodes in the distributed system, and the plurality of nodes being divided into one or more partitions. The method further comprises, in response to receiving a response to the heartbeat messages from a portion of slave nodes in the plurality of slave nodes, determining respective states of the one or more partitions. In addition, the method further comprises a state of a first slave node in the plurality of slave nodes at least based on the respective states of the one or more partitions, the master node failing to receive a response to the heartbeat messages from the first slave node.