Database Replication Manager with Dynamic Capacity Adjustment

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Database replication across multiple destinations introduces complexity and cost due to increased latency and processing requirements, particularly when performing multiple writes, which can reduce availability and necessitate custom functions.

Innovation Solution

Implementing a system that enables one-click replication of changes from a source database to a destination, with automatic capacity modification based on performance monitoring to maintain acceptable replication lag, using distributed database architecture and replication nodes to manage data format compatibility and order preservation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If data is replicated to multiple destinations, then data availability and access capabilities are improved, but system complexity and processing cost increase

Engineering Contradiction:

Improvedata access capabilitiesVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent introduces a replication manager as an intermediary component that coordinates data replication between the source database and multiple destinations. This mediator handles the complexity of multi-destination replication centrally, allowing clients to simply enable replication without managing the complex coordination required to maintain data consistency across multiple destinations, thus improving adaptability while controlling system complexity

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The replication manager provides universal functionality by handling replication to multiple different destination types (databases, data lakes, data warehouses) through a single unified interface. This multi-functional approach allows the system to support diverse data access capabilities without requiring separate customization for each destination type, reducing overall system complexity

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Reliability

If multiple writes are performed to different database services, then data consistency across destinations is improved, but database availability and latency are worsened

Engineering Contradiction:

Improvedata consistencyVSAvoidlatency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by having the replication manager proactively replicate data changes to all destinations in the background immediately after the source database write succeeds. This allows the client application to complete its write operation and receive confirmation quickly, while the consistency-maintaining replication operations proceed separately without blocking the client, thus improving reliability without significantly increasing perceived latency

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent segments the write operation into two independent parts: the primary write to the source database (which the client waits for) and the secondary replication writes to destination databases (handled asynchronously by the replication manager). This segmentation allows the client to achieve low latency by not waiting for replication, while the system maintains data consistency through the separate replication process

Inventive Principle:

Principle #1Segmentation

3Speed

If replication capacity is increased to reduce replication lag, then replication performance is improved, but resource consumption and cost increase

Engineering Contradiction:

Improvereplication speedVSAvoidresource consumption

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

The replication manager dynamically adjusts replication capacity based on monitored replication lag and system conditions. When replication lag exceeds thresholds, the system automatically increases replication capacity by allocating additional resources; when lag is acceptable, it reduces capacity to conserve resources. This dynamic adaptation allows the system to maintain good replication performance only when necessary, optimizing the balance between speed and resource consumption

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system implements feedback control by continuously monitoring replication lag and using this information to adjust replication capacity. The replication manager receives feedback about replication performance and automatically modifies resource allocation in response, ensuring that replication speed is increased only when actual lag conditions warrant it, thereby avoiding unnecessary resource consumption during periods of acceptable performance

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS11803568B1Replicating changes from a database to a destination and modifying replication capacity

Publication Date: 2023.10.31 AMAZON TECH INC

AI summary

A client of a distributed database service may conveniently enable replication of data from a source table to a destination (e.g., a different service/database table) by submitting a request enable replication for the source table. In response to the request to enable replication, the database service enables replication for the source table. To replicate a change, the database service may modify a format of the change based on a data storage format of the destination that is different than a data storage format of the database service. The database service monitors the performance of the replication and based on the performance, the database service may automatically modify the capacity to perform the replication of changes to the destination (e.g., by assigning additional resources to replication).