Adaptive Database Replication via Storage Layer Agent

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing database systems face challenges in distributing databases while maintaining ACID properties, leading to high costs and complexity, especially in scaling out access and keeping replicas synchronized across geographical distances, which results in significant replication lag.

Innovation Solution

Adaptive replication techniques that dynamically adjust to different scenarios by using a replication agent to selectively update database copies, minimizing downtime and network traffic by sending redo log records instead of data pages, and allowing the storage layer to handle changes and replication independently.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If database copies are updated by running SQL statements locally on each copy, then the database structure changes can be replicated, but the replication time increases and database copies fall further behind the source database

Engineering Contradiction:

Improvereplication synchronizationVSAvoidreplication time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent extracts the replication process from the database engine by introducing a separate replication agent that operates independently. The replication agent captures changes at the storage layer and propagates them to copies without requiring the database engine to execute SQL statements on each replica, thereby decoupling replication from query processing and reducing replication time

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The replication agent serves as an intermediary between the source database and its copies. Instead of the database engine directly managing replication through SQL statements, the replication agent mediates by capturing changes at the storage layer and applying them to copies, enabling more efficient and faster replication

Inventive Principle:

Principle #24Intermediary (Mediator)

2Ease of manufacture

If a shared disk model is used where every compute node has access to the same underlying data, then data sharing is simplified, but cache coherency management becomes complex and costly

Engineering Contradiction:

Improvedata sharing simplicityVSAvoidcache coherency management

Core Design Contradiction:

Ease of manufactureVSDevice complexity

Solution Approach 1:

The patent segments the database system into independent storage nodes that do not share disks. Each storage node maintains its own copy of the data, eliminating the need for complex cache coherency management while still enabling data sharing through the replication mechanism. This divides the system into independent units that can operate autonomously

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Instead of having multiple nodes share the same disk and manage cache coherency, the patent inverts the approach by having each node maintain its own data copy and using a replication agent to synchronize changes. This reverses the traditional shared disk model to a replicated storage model, simplifying the system architecture

Inventive Principle:

Principle #13The other way round (Inversion)

3Reliability

If databases are distributed across multiple nodes to provide fault tolerance, then reliability improves, but the system becomes more complex and costly to deploy and maintain

Engineering Contradiction:

Improvefault toleranceVSAvoidsystem deployment and maintenance complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The replication agent enables storage nodes to self-manage replication without requiring complex coordination between database engines. Each storage node can independently receive and apply changes through the replication agent, making the system easier to deploy and maintain while preserving fault tolerance through distribution

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The replication agent acts as an intermediary that simplifies distributed database management. Instead of requiring complex inter-node coordination for replication, the agent mediates by capturing changes at the storage layer and propagating them automatically, reducing deployment and maintenance complexity while maintaining reliability

Inventive Principle:

Principle #24Intermediary (Mediator)

4Productivity

If redo log records are sent instead of data pages for replication, then network traffic is minimized and replication performance improves, but the complexity of handling incremental changes increases

Engineering Contradiction:

Improvereplication speedVSAvoidincremental change handling complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent extracts the change tracking mechanism from the database engine by having the replication agent capture changes at the storage layer. This separates the incremental change handling from the database engine's transaction management, allowing efficient replication of only modified data through redo log records without increasing overall system complexity

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS10929428B1Adaptive database replication for database copies

Publication Date: 2021.02.23 AMAZON TECH INC

AI summary

Adaptive replication of changes may be performed for copies of a database. Log records may be generated and stored that correspond to changes to a database while a database is being copied. If the changes to be applied to a copy of the database is less than or equal to a threshold number of changes, then the copy of the database may be updated using the stored log records. If the changes to be applied to the copy of the database are greater than the threshold number of changes, then the copy of the database may be updated using data stored in the database.