I/O Path Reservation for Container Storage Split Brain Prevention

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing container platform technologies face challenges in preventing data corruption due to split brain scenarios, particularly in out-of-band managed storage systems where in-band SCSI reservation methods are not applicable, leading to potential data inconsistency and corruption when multiple containers write to the same volume.

Innovation Solution

Implementing a method to delete the I/O path between a container and a volume when launching a new container, and for storage orchestrator failures, deleting the path between the container and the quorum volume, while establishing a new I/O path between the new container and the secondary volume, thereby preventing simultaneous writes and maintaining data integrity.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If SCSI reservation commands are used to prevent split brain, then data consistency is improved, but the method is not applicable to out-of-band managed storage systems

Engineering Contradiction:

Improvedata consistencyVSAvoidapplicability to out-of-band systems

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent introduces a storage orchestrator as an intermediary component that manages I/O paths between containers and storage devices. The orchestrator receives container launch requests, identifies existing I/O paths, and coordinates path deletion to prevent split brain. This mediator approach replaces the direct SCSI reservation mechanism with an out-of-band management approach that is applicable to modern storage systems.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent substitutes the in-band SCSI reservation mechanism (which operates within the data I/O path) with an out-of-band I/O path management mechanism. Instead of using SCSI commands embedded in data operations, the system uses separate management interfaces to delete I/O paths, replacing the mechanical SCSI protocol with a software-based orchestration approach.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If the storage orchestrator deletes the I/O path, then exclusive access is ensured, but network failures or orchestrator errors may prevent path deletion

Engineering Contradiction:

Improveexclusive accessVSAvoidnetwork failures and orchestrator errors

Core Design Contradiction:

ReliabilityVSObject-affected harmful factors

Solution Approach 1:

The patent implements a timeout mechanism that acts as a cushion against potential failures. When the storage orchestrator requests I/O path deletion, it waits for a predetermined time period. If the deletion is not confirmed within this timeout period (indicating potential network failure or orchestrator error), the system automatically proceeds with alternative measures to ensure exclusive access, such as forcing path deletion or notifying other components of the failure condition.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Solution Approach 2:

The system takes preliminary anti-action by implementing failure detection and recovery mechanisms before split brain can occur. The timeout mechanism detects potential failures in the path deletion process, and the system prepares countermeasures (such as forcing deletion or notifying other nodes) to prevent the harmful effect of duplicate container access before it can compromise data consistency.

Inventive Principle:

Principle #9Preliminary anti-action

3Reliability

If I/O path deletion is implemented, then split brain is prevented, but the process complexity increases

Engineering Contradiction:

Improvesplit brain preventionVSAvoidpath management process

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent merges the I/O path management functionality into the existing storage orchestrator component. Instead of creating a separate dedicated system for path management, the orchestrator that already handles container-to-storage mapping is extended with path deletion capabilities. This combining of functions reduces overall system complexity while maintaining split brain prevention through centralized orchestration.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS11106371B1I/O path reservation with out of band management

Publication Date: 2021.08.31 HITACHI VANTARA LTD

AI summary

Example implementations described herein are directed to Input/Output (I/O) path reservation with out of band management. In example implementations, for failure of a storage orchestrator to delete the path between the container and the first volume, the example implementations described herein are directed to deleting a path between the first volume and a quorum volume; and establishing an I/O path between the new container and the second volume.