Pre-forking Replica Host Groups for Distributed Storage Scaling

Solution Overview

Problem

Existing systems require significant system resources and cause performance slowdowns during the duplication of replica hosts for dynamic resource partitioning, necessitating a lower-overhead solution for replicating replica host groups.

Innovation Solution

Implementing pre-forking of replicas before partitioning, where a duplicate leader replica and secondary replicas are created on the same node, allowing for efficient resource allocation and reduced overhead during partitioning and merging processes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If existing systems duplicate replica hosts during partitioning, then resource scaling is achieved, but system resource utilization increases and performance slowdowns occur

Engineering Contradiction:

Improveresource scalingVSAvoidsystem resource utilization

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary forking of replica hosts before partitioning is triggered. By creating duplicate replica hosts in advance while the original host is still active, the system prepares scaled resources without immediately consuming additional system resources for consensus operations. This preliminary action allows the system to have replica hosts ready for partitioning while avoiding the resource overhead of real-time consensus algorithms.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If consensus algorithms are used for leader election among duplicate replicas, then reliable leader selection is achieved, but performance slowdowns occur

Engineering Contradiction:

Improveleader selection reliabilityVSAvoidsystem performance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system performs preliminary forking of replica hosts before partitioning is triggered. By creating duplicate replica hosts in advance while the original host is still active, the system prepares scaled resources without immediately consuming additional system resources for consensus operations. This preliminary action allows the system to have replica hosts ready for partitioning while avoiding the resource overhead of real-time consensus algorithms.

Inventive Principle:

Principle #10Preliminary action

3Quantity of substance

If replica hosts are duplicated during partitioning, then storage capacity is increased, but device complexity increases

Engineering Contradiction:

Improvestorage capacityVSAvoidpartitioning process complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The system performs preliminary forking of replica hosts before partitioning is triggered. By creating duplicate replica hosts in advance while the original host is still active, the system prepares scaled resources without immediately consuming additional system resources for consensus operations. This preliminary action allows the system to have replica hosts ready for partitioning while avoiding the resource overhead of real-time consensus algorithms.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10712964B2Pre-forking replicas for efficient scaling of a distributed data storage system

Publication Date: 2020.07.14 AMAZON TECH INC

AI summary

A leader replica of a replica host group of a distributed storage service is configured to replicate the leader replica and its associated one or more secondary replicas to create a duplicate host group comprising a duplicate leader replica and one or more duplicate secondary replicas. The leader replica may appoint the duplicate leader replica or another host as the leader of the duplicate host group. The leader replica may assign a subset of database keys to be served by the duplicate host group and forward data requests directed a key of the subset of keys assigned to the duplicate host group. The leader replica may further update a global key table of the distributed data storage system to direct data requests associated with the subset of database keys to the duplicate leader replica.