Replica Latency Monitoring via Heartbeat Transactions

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In distributed computing environments, it is challenging to monitor and diagnose performance issues in log-first distributed databases due to their complex nature, leading to difficulties in pinpointing the source of performance degradation without introducing additional issues that further impact performance.

Innovation Solution

Implementing a system that uses service instances to submit 'heartbeat' transactions with timestamps to a log-first distributed database, allowing a metrics manager to calculate replica latency and perform actions such as removing or replacing hosts if latency exceeds a threshold, with minimal impact on performance, and sending notifications to clients.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Difficulty of detecting and measuring

If monitoring and diagnostics are implemented in distributed databases, then performance issue detection capability is improved, but system complexity and resource consumption increase

Engineering Contradiction:

Improveperformance issue detection capabilityVSAvoidsystem complexity

Core Design Contradiction:

Difficulty of detecting and measuringVSDevice complexity

Solution Approach 1:

The monitoring system is segmented into independent components: service instances that generate heartbeat transactions, a transaction log that records transactions, and a metrics manager that analyzes performance data. This segmentation allows each component to perform its function independently without increasing overall system complexity

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces heartbeat transactions as an intermediary mechanism to indirectly measure replica performance. Instead of directly monitoring replica operations, the system uses these intermediary transactions to gauge performance through latency measurements, simplifying the detection process

Inventive Principle:

Principle #24Intermediary (Mediator)

2Difficulty of detecting and measuring

If monitoring transactions are submitted to the database, then performance monitoring capability is improved, but network bandwidth and computing resources are consumed

Engineering Contradiction:

Improveperformance monitoring capabilityVSAvoidnetwork bandwidth and computing resource consumption

Core Design Contradiction:

Difficulty of detecting and measuringVSLoss of energy

Solution Approach 1:

The system uses a minimal set of heartbeat transactions rather than comprehensive monitoring of all database operations. These partial monitoring actions provide sufficient performance data while consuming minimal network bandwidth and computing resources

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The heartbeat transactions are designed as lightweight, short-lived objects that are quickly created and processed. Each heartbeat transaction is simple in structure and is discarded after serving its measurement purpose, minimizing resource consumption

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

3Difficulty of detecting and measuring

If additional monitoring components are added to distributed databases, then performance diagnostics are improved, but application performance may be further degraded

Engineering Contradiction:

Improveperformance diagnosticsVSAvoidapplication performance

Core Design Contradiction:

Difficulty of detecting and measuringVSProductivity

Solution Approach 1:

The monitoring functionality is merged with existing database components rather than adding separate monitoring infrastructure. Service instances that already exist in the system are used to generate heartbeat transactions, and the transaction log serves dual purposes for both data storage and performance monitoring

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The system uses its own existing infrastructure to perform monitoring. Service instances monitor their own performance by submitting heartbeat transactions, and the database's transaction log automatically records these transactions without requiring external monitoring tools

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11487738B1Replica performance for transactions submitted to log-first distributed databases

Publication Date: 2022.11.01 AMAZON TECH INC

AI summary

By obtaining metadata for transactions submitted by a service to a log-first distributed database of a provider network, a metrics manager may determine database replica performance for those transactions and notify clients of potential performance issues. When an instance of the service submits a write transaction to the log-first distributed database, the transaction may include a host name and a timestamp for the submission of the transaction. At a later point in time, a write applier may obtain the transaction and apply it to a local database replica, along with an additional timestamp for the application of the transaction to the replica. A metrics manager may obtain the transaction timestamps from the replica and calculate a latency metric for the propagation of the transaction from the particular service instance/instance host to the replica. The latency metric may be stored or transmitted to an endpoint (e.g., a client or administrator).