Distributed Storage Metadata Availability During Node Outages

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing distributed storage systems face challenges in maintaining metadata integrity and availability across potentially unreliable nodes, leading to issues with data accessibility during multi-node outages.

Innovation Solution

The system implements a metadata management scheme that allows the cluster to operate in a read-only mode when regions are missing, promoting backup copies to authoritative status and ensuring data accessibility by maintaining a region map and synchronizing metadata across nodes, enabling continued read operations even with missing regions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If metadata is distributed across unreliable nodes in a distributed storage system, then system scalability and flexibility are improved, but metadata integrity and availability deteriorate during node outages

Engineering Contradiction:

Improvesystem scalabilityVSAvoidmetadata availability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system performs preliminary actions by creating backup copies of metadata regions before node failures occur. Each metadata region is replicated across multiple nodes with designated primary and backup copies, ensuring that metadata availability is maintained even when nodes fail during operation.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system provides beforehand cushioning against node failures by implementing redundant metadata storage. Backup metadata copies are prepared in advance on separate nodes, cushioning the system against the loss of primary metadata during unexpected node outages and preventing complete metadata unavailability.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

2Stability of the object's composition

If the system maintains strict metadata consistency across all nodes, then data integrity is improved, but system availability during outages deteriorates

Engineering Contradiction:

Improvemetadata consistencyVSAvoidsystem availability

Core Design Contradiction:

Stability of the object's compositionVSReliability

Solution Approach 1:

The system applies local quality by allowing different nodes to have different metadata states during outages. The primary metadata region maintains strict consistency, while backup regions on other nodes can remain in a stale but accessible state, enabling read operations to continue locally even when the primary node is unavailable.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system implements partial action by allowing read operations to proceed with backup metadata copies even when full consistency cannot be maintained. This partial operation capability ensures system availability during outages, accepting that some metadata may be stale until the primary node recovers and synchronizes the backups.

Inventive Principle:

Principle #16Partial or excessive action

3Reliability

If the system enters a read-only mode during metadata outages, then data protection is improved, but operational flexibility deteriorates

Engineering Contradiction:

Improvedata protectionVSAvoidoperational flexibility

Core Design Contradiction:

ReliabilityVSEase of operation

Solution Approach 1:

The system applies segmentation by dividing the metadata storage into independent regions with primary and backup copies. During outages, only the affected regions enter read-only mode while other regions remain fully operational, maintaining operational flexibility for unaffected data while protecting the affected regions.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system implements partial action by allowing write operations to continue for metadata regions that are not affected by outages. Only the specific regions experiencing metadata loss enter read-only mode, while the rest of the system maintains full operational flexibility, enabling partial system functionality during outages.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS9904605B2System and method for enhancing availability of a distributed object storage system during a partial database outage

Publication Date: 2018.02.27 HITACHI VANTARA LLC

AI summary

An “operate with missing region” feature allows a cluster to continue servicing reads for available regions even when some regions are missing. Upon a given node failure condition, the cluster is placed in an effective read-only mode for all regions. The node failure condition typically is one where there has been a failure of an authoritative region copy and no backup copy is then available. Typically, “read-only” means no client write or update requests succeed while the cluster is in this state. Those requests should fail with an error message that indicates failure because of the read-only status. Preferably, such requests are re-tried. In this mode, regions are only allowed to perform read operations but the cluster continues to operate with missing regions, and missing regions are entered on a region map. The cluster then automatically recovers returning missing regions, after which it leaves the read-only state.