Replication Engine for Distributed File Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Modern distributed file systems face challenges in establishing suitable replication schedules due to their large size and asynchronous state changes, making it difficult to manage replication and backup operations effectively.

Innovation Solution

A replication engine is instantiated to establish a secure communication channel between source and target file systems using a handshake protocol with one-time-secret authentication, and to execute replication jobs based on rules and metrics such as processor and network utilization, with the ability to delay or pause jobs during blackout periods.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If continuous replication is implemented in distributed file systems, then data redundancy and availability are improved, but system complexity and difficulty of managing replication schedules increase

Engineering Contradiction:

Improvedata availabilityVSAvoidreplication management complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The replication engine automatically monitors file system changes and executes replication jobs without manual intervention. The system self-manages the replication schedule by continuously monitoring for changes and autonomously determining when replication should occur, eliminating the need for complex external scheduling mechanisms.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system continuously monitors file system state changes and uses this feedback to trigger replication operations. The replication engine receives feedback about changes from the file system and automatically adjusts replication execution based on this information, creating a closed-loop control system that simplifies management.

Inventive Principle:

Principle #23Feedback

2Reliability

If replication jobs are executed frequently to ensure data consistency, then data integrity is improved, but resource utilization and system performance deteriorate

Engineering Contradiction:

Improvedata consistencyVSAvoidsystem resource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

Instead of continuous monitoring and replication, the system uses periodic replication triggered by file system events. Replication occurs at specific intervals or upon detecting changes, rather than continuously, reducing resource consumption while maintaining data consistency.

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

The replication frequency and resource allocation are dynamically adjusted based on system conditions. The replication engine can pause or delay jobs during high-load periods and resume when resources are available, optimizing the balance between data consistency and resource utilization.

Inventive Principle:

Principle #15Dynamics

3Reliability

If replication operations are performed continuously, then data synchronization is improved, but interference with critical operations increases

Engineering Contradiction:

Improvedata synchronizationVSAvoidinterference with critical operations

Core Design Contradiction:

ReliabilityVSObject-generated harmful factors

Solution Approach 1:

The system implements preliminary checks and conditions before executing replication operations. It monitors system state and prevents replication from interfering with critical operations by establishing conditions that must be met before replication proceeds, such as checking for active critical operations and delaying replication if necessary.

Inventive Principle:

Principle #9Preliminary anti-action

Solution Approach 2:

The replication engine acts as an intermediary between file system changes and the target file system. It monitors changes and intelligently determines when and how to replicate, mediating between the need for synchronization and the need to avoid interfering with critical operations.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Reliability

If secure communication channels are established using handshake protocols, then security is improved, but establishment time and complexity increase

Engineering Contradiction:

ImprovesecurityVSAvoidchannel establishment time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Security credentials and authentication information are pre-configured and stored in the file system metadata before replication begins. The handshake protocol uses these pre-established credentials to quickly authenticate and establish secure channels, eliminating the need for time-consuming key exchange procedures during replication operations.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10606812B2Continuous replication for secure distributed filesystems

Publication Date: 2020.03.31 QUMULO INC

AI summary

Embodiments are directed to file systems. A replication engine may establish a secure communication channel between a source file system and a target file system. The replication engine may: instantiate a replication job associated with rules; determine changes in the source file system; determine characteristics of the replication job that may be based on the changes; compare the to the characteristics and a black-out schedule; execute the replication job to communicate the changes in the source file system to the target file system based on a result of the comparison. Upon completion of the replication job, the replication engine may automatically instantiating a next replication job to copy subsequent changes in the source file system to the target file system.