Auditor-Based Grey Failure Handling in Distributed Storage

Solution Overview

Problem

Distributed database storage systems across multiple availability zones face challenges in handling failures, leading to unnecessary data replication that can cause bandwidth and resource issues, especially in cases of temporary or unclear failures, where delaying replication can help determine the permanence of the failure.

Innovation Solution

Implementing an auditor that assesses the health of storage servers across availability zones, using low and high watermark thresholds along with a gray area to determine when to initiate or delay data replication, thereby avoiding unnecessary data transfer and maintaining data durability.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is replicated immediately upon detecting a failure in an availability zone, then data durability is improved, but bandwidth and resource utilization worsen due to unnecessary transfers during temporary failures

Engineering Contradiction:

Improvedata durabilityVSAvoidbandwidth and resource utilization

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The system performs preliminary assessment of failure permanence before initiating data replication. The auditor evaluates whether a failure is temporary or permanent using watermark thresholds, and only triggers replication when the failure is confirmed permanent, thereby avoiding premature replication actions during transient failures

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

An auditor component is introduced as an intermediary between failure detection and data replication. The auditor assesses the permanence of failures using watermark thresholds and determines whether replication should be triggered, acting as a mediator that filters out unnecessary replication events while ensuring durability when needed

Inventive Principle:

Principle #24Intermediary (Mediator)

2Loss of energy

If data replication is delayed to assess failure permanence, then bandwidth and resource utilization is improved, but data durability may be compromised if replication is delayed too long

Engineering Contradiction:

Improvebandwidth and resource utilizationVSAvoiddata durability

Core Design Contradiction:

Loss of energyVSReliability

Solution Approach 1:

The system performs preliminary assessment of failure permanence before initiating data replication. The auditor evaluates whether a failure is temporary or permanent using watermark thresholds, and only triggers replication when the failure is confirmed permanent, thereby avoiding premature replication actions during transient failures

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

An auditor component is introduced as an intermediary between failure detection and data replication. The auditor assesses the permanence of failures using watermark thresholds and determines whether replication should be triggered, acting as a mediator that filters out unnecessary replication events while ensuring durability when needed

Inventive Principle:

Principle #24Intermediary (Mediator)

3Loss of energy

If watermark thresholds are used to determine replication timing, then unnecessary data transfers are reduced, but system complexity increases due to threshold management

Engineering Contradiction:

Improvebandwidth and resource utilizationVSAvoidsystem complexity

Core Design Contradiction:

Loss of energyVSDevice complexity

Solution Approach 1:

The system uses watermark thresholds (high watermark and low watermark) as parameters to control the replication decision process. By changing the state variable from binary (failure detected/replicate) to continuous (availability percentage between thresholds), the system reduces unnecessary replication while managing complexity through configurable parameters

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11622000B2Grey failure handling in distributed storage systems

Publication Date: 2023.04.04 SALESFORCE INC

AI summary

Techniques are disclosed relating to managing distributed storage of data across availability zones and the replication of data in case of storage server failures. A distributed storage system may include storage servers distributed across availability zones with an auditor instantiated in at least one storage server. The auditor manages the replication of data in the event of one or more storage servers failure in an availability zone. In the event of the failure, the auditor may determine the extent of the failure and whether the failure involves a small number or a large number of storage servers. In the event a large number of storage servers being affected, the auditor may delay replication of data to see if the failure is temporary and avoid unwanted data transfer of large amounts of data across availability zones.