Epoch-Based File System Replication Engine

Solution Overview

Problem

Modern distributed file systems face challenges in efficiently managing data replication due to the complexity of determining which file system objects need to be replicated, leading to significant storage and performance overheads.

Innovation Solution

A replication engine is instantiated to traverse the file system, using epoch values and snapshot information to identify and replicate only the necessary file system objects, thereby optimizing replication operations by excluding mismatched epochs and communicating relevant blocks to a target file system.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If naive backup strategies are used to replicate all file system objects, then data redundancy and availability are improved, but storage overhead and performance degradation occur

Engineering Contradiction:

Improvedata availabilityVSAvoidstorage overhead

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent extracts only the changed file system objects for replication by comparing epoch values between source and target file systems. Instead of copying all objects, the replication engine identifies and replicates only those objects whose epoch values indicate modifications, thereby reducing storage overhead while maintaining data availability.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent implements partial replication by performing backup operations only on the subset of file system objects that have changed. The replication engine determines which objects need replication by comparing epoch values, executing replication actions only on necessary objects rather than all objects, thus optimizing storage resource usage.

Inventive Principle:

Principle #16Partial or excessive action

2Reliability

If naive backup strategies are used to replicate all file system objects, then data redundancy is improved, but performance overhead increases

Engineering Contradiction:

Improvedata redundancyVSAvoidreplication performance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent extracts only the changed file system objects for replication by comparing epoch values between source and target file systems. Instead of copying all objects, the replication engine identifies and replicates only those objects whose epoch values indicate modifications, thereby reducing storage overhead while maintaining data availability.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent implements partial replication by performing backup operations only on the subset of file system objects that have changed. The replication engine determines which objects need replication by comparing epoch values, executing replication actions only on necessary objects rather than all objects, thus optimizing storage resource usage.

Inventive Principle:

Principle #16Partial or excessive action

3Measurement precision

If epoch comparison is performed on all file system objects, then replication accuracy is improved, but computational complexity increases

Engineering Contradiction:

Improvereplication accuracyVSAvoidcomputational complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent segments the file system into individual objects, each with an associated epoch value. The replication engine traverses the file system object by object, comparing epoch values independently for each object. This segmentation allows precise identification of changed objects without requiring complex global analysis, reducing computational complexity while maintaining replication accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements partial replication by performing backup operations only on the subset of file system objects that have changed. The replication engine determines which objects need replication by comparing epoch values, executing replication actions only on necessary objects rather than all objects, thus optimizing storage resource usage.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11151092B2Data replication in distributed file systems

Publication Date: 2021.10.19 QUMULO INC

AI summary

Embodiments are directed to a replication engine that provides a root node for a file that may be associated with a replication snapshot where the file is comprised of the root node, internal nodes, and data nodes. The replication engine may start at the root node and traverse the file system. If a visited internal node has an epoch value that matches the epoch, the replication engine may continue the traversal by visiting a next unvisited adjacent internal node. If the visited internal node has an epoch value that mismatches the epoch, the replication engine may backtrack to a nearest adjacent internal node that matches the epoch. And, in response to visiting a data node the replication engine may be arranged to determine blocks that are associated with the data node and communicate a copy of the blocks to a target file system.