Database Transaction Batching via Global Timestamping

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In distributed relational database management systems (RDBMS), the use of a central global transaction manager (GTM) for multi-version concurrency control (MVCC) leads to network bandwidth and CPU overload due to the need for separate requests for each transaction, causing performance bottlenecks.

Innovation Solution

Implementing a method where multiple requests are combined into a single batch request using a pool of persistent connections to the GTM server, minimizing lock contention and reducing the load on the GTM server by sending only one request over one connection, and distributing the results to satisfy database requests efficiently.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If separate requests are sent for each transaction to the GTM server, then data validity and isolation are guaranteed, but network bandwidth and GTM server CPU become overwhelmed causing performance bottlenecks

Engineering Contradiction:

Improvedata validityVSAvoidperformance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

Multiple separate transaction requests are merged into a single batch request that is sent to the GTM server. The batch request includes multiple snapshot requests and commit requests, which are processed together in one network round-trip. This merging reduces the number of network communications and GTM server CPU cycles required, thereby improving performance while still ensuring data validity through proper batch processing and timestamp assignment.

Inventive Principle:

Principle #5Merging (Combining)

2Productivity

If multiple persistent connections are used to the GTM server, then request handling capacity increases, but connection management complexity and resource usage increase

Engineering Contradiction:

Improverequest handling capacityVSAvoidconnection management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system dynamically adjusts the number of persistent connections to the GTM server based on the workload and batch size requirements. Rather than using a fixed large number of connections, the system creates connections as needed and manages their lifecycle dynamically. This allows the system to scale connection usage according to actual demand, improving request handling capacity while avoiding the overhead of maintaining a permanently large connection pool.

Inventive Principle:

Principle #15Dynamics

3Productivity

If batch requests are combined into single requests, then network bandwidth usage and GTM server load are reduced, but lock contention and request processing complexity increase

Engineering Contradiction:

Improvenetwork efficiencyVSAvoidrequest processing complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The batch request processing is segmented into distinct phases: request collection, batch assembly, GTM server communication, and result distribution. Each phase is handled by dedicated code modules that process specific aspects of the batch operation. This segmentation reduces lock contention by minimizing the scope and duration of critical sections, while also making the overall complex process more manageable through clear separation of concerns.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

An intermediary batch processing layer is introduced between the individual transaction requests and the GTM server. This intermediary collects requests, manages the batch assembly, handles the single network communication, and then distributes results back to the original requestors. This intermediary layer simplifies the complexity by providing a clear interface and management point, rather than having each request directly interact with the GTM server individually.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Reliability

If a central GTM server is used for MVCC, then transaction isolation is achieved, but the GTM server becomes a single point of failure and performance bottleneck

Engineering Contradiction:

Improvetransaction isolationVSAvoidsingle point of failure

Core Design Contradiction:

ReliabilityVSObject-affected harmful factors

Solution Approach 1:

The system performs preliminary actions by establishing persistent connections to the GTM server in advance and maintaining them in a ready state. Batch requests are prepared and queued locally before being sent to the GTM server, reducing the need for frequent connection establishment and teardown. This preliminary preparation reduces the load on the GTM server and minimizes the impact of potential failures, as the system can buffer and retry batches rather than losing individual transaction state.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12130796B2Method and apparatus for distributed database transactions using global timestamping

Publication Date: 2024.10.29 HUAWEI TECH CO LTD

AI summary

A method and system for implementing concurrency control in a database system includes receiving a request from a node for handling by a GTM. Registering the request in a slot in a tracking array where each of the slots represents one of the requests for handling by the GTM. A slot is acquired, and a segment of the tracking array is scanned to determine a batch of pending requests. The batch is used to produce an aggregate request and the aggregate request includes an aggregate increment of a total number of commit requests of the batch. The aggregate request is sent to the GTM using one of a plurality of pre-established connections. A response is received from the GTM that includes a current value of an incremented timestamp. The incremented timestamp is distributed to the requests of the batch and the lock is released.