Non-disruptive Planned Failover in Distributed Storage Clusters

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Multi-site distributed data storage systems face disruptions due to network failures and lack of non-disruptive planned failover capabilities without an external mediator, leading to inefficiencies and data consistency issues.

Innovation Solution

Implementing a planned failover feature within the primary storage cluster as an authority, using a state machine with a persistent configuration database to track the failover state and ensure non-disruptive operations by designating roles and using atomic test and set procedures to avoid split-brain situations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If an external mediator is used to coordinate failover between primary and secondary storage clusters, then data consistency and coordination reliability are improved, but device complexity and system cost increase

Engineering Contradiction:

Improvecoordination reliabilityVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent removes the external mediator component from the failover coordination system. Instead of having a separate mediator service, the primary storage cluster itself performs failover coordination using its existing quorum architecture, thereby eliminating device complexity while maintaining reliability

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The primary storage cluster performs self-coordination for failover operations using its internal quorum mechanism. The cluster members vote among themselves to determine failover timing and execution, eliminating the need for external mediator services and reducing system complexity

Inventive Principle:

Principle #25Self-service

2Productivity

If failover operations are executed without disrupting ongoing I/O operations, then service continuity and user experience are improved, but control complexity and coordination difficulty increase

Engineering Contradiction:

Improveservice continuityVSAvoidcontrol complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system performs preliminary actions by draining in-flight I/O operations before initiating the failover process. This ensures that no active I/O operations are present during the failover transition, enabling non-disruptive failover while maintaining manageable control complexity through a clear sequential operation model

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The failover process is segmented into distinct phases: draining in-flight operations, obtaining quorum votes from cluster members, executing the failover, and resuming operations. This segmentation simplifies control by breaking down the complex failover task into manageable, ordered steps

Inventive Principle:

Principle #1Segmentation

3Adaptability or versatility

If a timing window allows both primary and secondary copies to claim master role simultaneously, then failover flexibility and readiness are improved, but data consistency and system stability deteriorate due to split-brain scenarios

Engineering Contradiction:

Improvefailover flexibilityVSAvoiddata consistency

Core Design Contradiction:

Adaptability or versatilityVSStability of the object's composition

Solution Approach 1:

The system uses feedback mechanisms in the form of quorum votes from cluster members to determine when failover should proceed. Each cluster member votes based on its current state and readiness, providing feedback that prevents simultaneous master role claims and maintains data consistency while allowing flexible failover timing

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system performs preliminary quorum voting and readiness assessment before allowing the timing window to open. This preliminary action ensures that only one cluster can claim master role at a time, preventing split-brain scenarios while maintaining failover flexibility through coordinated timing

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11704207B2Methods and systems for a non-disruptive planned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system without using an external mediator

Publication Date: 2023.07.18 NETAPP INC

AI summary

Systems and methods are described for a non-disruptive planned failover from a primary copy of data at a primary storage cluster to a mirror copy of the data at a cross-site secondary storage cluster without using an external mediator. According to an example, a planned failover feature of a multi-site distributed storage system provides an order of operations such that a primary copy of a first data center continues to serve I/O operations until a mirror copy of a second data center is ready. This planned failover feature improves functionality and efficiency of the distributed storage system by providing non-disruptiveness during planned failover without using an external mediator based on a primary storage cluster being selected as an authority to implement a state machine with a persistent configuration database to track a planned failover state for the planned failover.