Distributed Object Replication Architecture Dynamic Scaling

Solution Overview

Problem

Current data replication systems are unable to adapt to changing requirements, leading to inefficiencies such as over- or under-provisioning of resources, resulting in either wasteful resource allocation or unacceptably long replication times.

Innovation Solution

A distributed object replication system that dynamically allocates and deallocates worker nodes based on factors like bandwidth, IOPS, and computing costs, using a microservices architecture to efficiently replicate data across multiple sites, including cloud and non-cloud environments.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If resources are over-provisioned for data replication, then replication speed is improved, but resource waste increases

Engineering Contradiction:

Improvereplication speedVSAvoidresource waste

Core Design Contradiction:

SpeedVSLoss of energy

Solution Approach 1:

The system dynamically adjusts the number of worker nodes allocated to replication tasks based on real-time monitoring of replication progress, data volume, and system load. This allows the system to provision resources flexibly - scaling up worker nodes when replication speed needs to be improved and scaling down when the same replication can be achieved with fewer resources, thereby resolving the contradiction between replication speed and resource waste

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system implements continuous feedback loops that monitor replication task performance and system resource utilization. Based on this feedback, the system automatically adjusts resource allocation by adding or removing worker nodes from the pool, ensuring that resources are optimized for each replication task rather than being statically over-provisioned

Inventive Principle:

Principle #23Feedback

2Loss of energy

If resources are under-provisioned for data replication, then resource efficiency is improved, but replication time increases

Engineering Contradiction:

Improveresource efficiencyVSAvoidreplication time

Core Design Contradiction:

Loss of energyVSLoss of time

Solution Approach 1:

The system dynamically adjusts the number of worker nodes allocated to replication tasks based on real-time monitoring of replication progress, data volume, and system load. This allows the system to provision resources flexibly - scaling up worker nodes when replication speed needs to be improved and scaling down when the same replication can be achieved with fewer resources, thereby resolving the contradiction between replication speed and resource waste

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system implements continuous feedback loops that monitor replication task performance and system resource utilization. Based on this feedback, the system automatically adjusts resource allocation by adding or removing worker nodes from the pool, ensuring that resources are optimized for each replication task rather than being statically over-provisioned

Inventive Principle:

Principle #23Feedback

3Device complexity

If fixed resources are allocated for replication, then system simplicity is maintained, but adaptability to changing requirements deteriorates

Engineering Contradiction:

Improvesystem simplicityVSAvoidadaptability to changing requirements

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

The system transitions from static resource allocation to dynamic resource provisioning, where the number of worker nodes is automatically adjusted based on replication task requirements and system conditions. This dynamic approach enables the system to adapt to changing data volumes, replication frequencies, and performance requirements without manual intervention, resolving the contradiction between simplicity and adaptability

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system implements self-service capabilities where the replication management system automatically monitors its own performance and adjusts resource allocation without external intervention. The system can autonomously add or remove worker nodes from the replication pool based on monitored metrics, maintaining operational simplicity while achieving high adaptability to changing requirements

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS12197758B2Distributed object replication architecture

Publication Date: 2025.01.14 EMC IP HLDG CO LLC

AI summary

A source site includes a controller, a set of source worker nodes, and a message queue connected between the controller and source worker nodes. The message queue receives messages and stores the messages for retrieval. A destination site includes a set of destination worker nodes. Tasks are generated to replicate changes to objects at the source site to the destination site. The controller pushes messages corresponding to the tasks onto the message queue. A source worker node retrieves a message corresponding to a task from the message queue for processing in conjunction with a destination worker node. The message is indicated as having been retrieved from the message queue.