Dependency-Aware Parallel Splitting for Metadata Replication

Solution Overview

Problem

Existing data replication methods, particularly synchronous replication, face increased latency and inconsistency issues due to sequential dispatching of metadata operations, which affects the transition to a synchronous replication state and client I/O performance.

Innovation Solution

Implementing dependency-aware parallel splitting of operations, where independent metadata operations are transmitted in parallel and dependent operations are executed sequentially to maintain consistency, reducing replication latency and preserving dependencies among metadata operations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If sequential dispatching of metadata operations is used to maintain consistency, then data consistency is improved, but replication latency increases

Engineering Contradiction:

Improvedata consistencyVSAvoidreplication latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent segments metadata operations into independent and dependent categories. Independent operations that do not affect the same inodes are identified and can be dispatched in parallel, while dependent operations are maintained in sequential order. This segmentation allows the system to exploit parallelism where safe and maintain consistency where necessary.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements a dynamic dispatching mechanism that adapts the execution order of metadata operations based on their dependencies. The system dynamically determines which operations can be executed in parallel and which must be sequential, optimizing the balance between consistency and latency for each specific set of operations.

Inventive Principle:

Principle #15Dynamics

2Productivity

If parallel transmission of metadata operations is used to reduce latency, then replication speed is improved, but data inconsistency may occur

Engineering Contradiction:

Improvereplication speedVSAvoiddata consistency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent segments metadata operations into independent and dependent categories. Independent operations that do not affect the same inodes are identified and can be dispatched in parallel, while dependent operations are maintained in sequential order. This segmentation allows the system to exploit parallelism where safe and maintain consistency where necessary.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system uses feedback mechanisms to track the execution status and dependencies of metadata operations. By monitoring which operations have been executed and their impact on inodes, the system can dynamically adjust the dispatching of subsequent operations to maintain consistency while maximizing parallelism.

Inventive Principle:

Principle #23Feedback

3Reliability

If synchronous replication is implemented to ensure data protection, then reliability is improved, but client I/O performance deteriorates

Engineering Contradiction:

Improvedata protectionVSAvoidclient I/O performance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent implements a dynamic dispatching mechanism that adapts the execution order of metadata operations based on their dependencies. The system dynamically determines which operations can be executed in parallel and which must be sequential, optimizing the balance between consistency and latency for each specific set of operations.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system performs preliminary analysis of metadata operation dependencies before dispatching them to the secondary node. By pre-identifying independent operations, the system can prepare parallel execution paths in advance, reducing the overall time required for synchronous replication without compromising data protection.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20230081436A1Dependency aware parallel splitting of operations

Publication Date: 2023.03.16 NETAPP INC

AI summary

Techniques are provided for dependency aware parallel splitting of operations. For example, a count of pending data operations being executed by a first node and replicated in parallel to a second node are tracked. A metadata operation is executed at the first node based upon the count being less than a threshold (e.g., the count being zero). A first list of affected inodes modified by the metadata operation is identified. A dependency of the metadata operation with respect to pending metadata operations replicated to the second node is determined. The metadata operation is dispatched to the second node based upon the dependency indicating that the metadata operation is independent of the pending metadata operations.