Cluster Fail-Over via Master Node Connection Release
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In container-based virtualization environments, existing fail-over technologies face challenges in safely managing permanent data during network failures, leading to data damage and recognition issues when the fail-over source node is restored.
Innovation Solution
A cluster system with a master node and two nodes connected via different networks, where the master node issues connection release instructions and manages post-processing through a second network, ensuring safe fail-over by updating connection states and deleting relevant files and mount point information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Duration of action of moving object
If the first node continues to perform write processing on the storage device through the second network during network failure, then the application remains operational, but data damage occurs due to concurrent writes from both the first node and second node
Solution Approach 1:
The master node detects network failure of the first node and proactively issues a connection release instruction to the storage device through the second network before the first node can cause data damage. This preliminary action prevents the harmful concurrent write scenario by releasing the first node's connection to the storage device volume.
Solution Approach 2:
The master node acts as an intermediary between the first node and the storage device. It monitors the network status of the first node and mediates the connection state to the storage device by issuing connection release instructions when failure is detected, thereby preventing data damage without requiring direct communication between the first node and storage device during failure.
2Speed
If the connection between the first node and the storage device is not released during network failure, then the first node can restore operations quickly, but post-processing cannot be executed and volume recognition fails upon restoration
Solution Approach 1:
The master node issues a connection release instruction to the storage device through the second network during the network failure period, performing the necessary post-processing preparation in advance. This allows the first node to quickly restore operations while ensuring that the storage device is in the correct state for volume recognition upon restoration.
Solution Approach 2:
The system implements feedback through the master node monitoring the network status of the first node and adjusting the connection state to the storage device accordingly. When network failure is detected, the master node provides feedback by issuing connection release instructions, ensuring the storage device state reflects the actual operational status of the first node.
3Device complexity
If a single network is used for all node communications, then the system structure is simplified, but network failures isolate nodes from master node and other nodes, preventing proper fail-over control
Solution Approach 1:
The network communication is segmented into two separate networks: a first network for direct node-to-node communication and a second network for master node control communications. This segmentation allows the master node to maintain control capability through the second network even when the first network fails, enabling reliable fail-over control while maintaining relatively simple network structures.
Solution Approach 2:
The system adds another dimension to network communication by introducing a second network for master node control alongside the first network for data communication. This dimensional addition provides redundancy and control capability during failures without significantly increasing overall system complexity.
Data Source
AI summary
A cluster system has nodes connected by a first network, including a master node, a first node performing an IO request with respect to a first volume of a storage device capable of including a plurality of volumes through a second network, and a second node taking over processing of the first node at the time of a failure of the first node. When a failure of the first network occurs in the first node, the master node transmits a connection release instruction through the second network. When the connection release instruction is received, the storage device releases the connection between the first node and the first volume, and updates connection state management information for managing such a connection state. The first node acquires the connection state to be managed through the second network, and executes post-processing relevant to the first volume that is associated with the connection release instruction.