Log Pusher for Distributed Data Replication Groups

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing computing systems face challenges in maintaining data consistency and availability during overload or failure scenarios, leading to potential data loss and reduced customer experience, while replicating data across multiple systems increases resource costs and latency.

Innovation Solution

Implementing a data replication group with a consensus protocol like Paxos, where nodes generate logs of operations and heartbeats, and a log pusher collects and processes these logs to publish them as a data stream, enabling event-driven processes and allowing clients to reconstruct the data replication group's state and history.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is replicated across multiple computer systems to protect against failures, then data durability and availability are improved, but resource costs and latency increase

Engineering Contradiction:

Improvedata durabilityVSAvoidresource costs

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The system segments data replication into primary and replica nodes, where primary nodes handle write operations and replica nodes handle read operations. This segmentation allows the system to maintain data durability through replication while optimizing resource utilization by distributing workloads across different node types.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a log pusher as an intermediary component that collects logs from primary nodes and distributes them to replica nodes. This intermediary mechanism streamlines the replication process, reducing the direct communication overhead between primary and replica nodes, thereby lowering latency and resource costs.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If data is replicated across multiple computer systems to protect against failures, then data durability and availability are improved, but customer experience deteriorates due to delay

Engineering Contradiction:

Improvedata availabilityVSAvoidcustomer experience delay

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by maintaining replica nodes that are pre-configured and ready to take over immediately in case of failure. Logs are continuously collected and prepared for distribution, so when a primary node fails, replicas can quickly assume its role without significant delay, improving data availability and customer experience.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The log pusher operates continuously to collect and distribute logs, ensuring that replica nodes are always up-to-date with the latest data changes. This continuous operation eliminates gaps in data synchronization, allowing the system to maintain high availability and respond quickly to customer requests even during failure scenarios.

Inventive Principle:

Principle #20Continuity of useful action

3Stability of the object's composition

If consensus protocol is used to maintain consistent replicas, then data consistency is improved, but system complexity increases

Engineering Contradiction:

Improvedata consistencyVSAvoidsystem complexity

Core Design Contradiction:

Stability of the object's compositionVSDevice complexity

Solution Approach 1:

The patent extracts the consensus protocol implementation from the core application logic and implements it as a separate, dedicated mechanism within the data replication group. This extraction allows the consensus protocol to be optimized independently and reduces the complexity burden on individual nodes, as they only need to participate in consensus rather than implement the full protocol themselves.

Inventive Principle:

Principle #2Taking out (Extraction)

4Stability of the object's composition

If service requests are denied to protect computer systems from overload, then system stability is improved, but customer experience deteriorates

Engineering Contradiction:

Improvesystem stabilityVSAvoidcustomer experience

Core Design Contradiction:

Stability of the object's compositionVSEase of operation

Solution Approach 1:

Instead of denying service entirely during overload conditions, the system applies partial action by allowing read operations to proceed on replica nodes while potentially throttling write operations to primary nodes. This approach maintains system stability by protecting primary nodes from overload while still providing customers with access to data through replicas, thereby preserving customer experience.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11640410B1Distributed log processing for data replication groups

Publication Date: 2023.05.02 AMAZON TECH INC

AI summary

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. The nodes of the data replication groups may generate logs containing information corresponding committed operations performed by the nodes. These logs may be collected and processed to obtain useful information corresponding to the operation of the data replication group. Furthermore, this processed information may be provided in the form of a stream to enable event driven operations corresponding to the logs.