Distributed Transaction Manager for MPP Database Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Centralized transaction managers in massively parallel processing (MPP) database systems face issues of single point of failure and limited scalability, which can jeopardize high availability and reliability, especially when handling snapshots for distributed transactions.
Innovation Solution
A distributed transaction manager is implemented, where each node maintains local snapshot information, supporting repeatable read isolation level and using an implicit two-phase commit protocol to ensure atomicity and durability, eliminating the need for a centralized component.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a centralized transaction manager is used to track snapshots globally for all nodes, then global transaction visibility is achieved, but single point of failure and limited scalability occur
Solution Approach 1:
The centralized transaction manager is segmented into distributed transaction managers across multiple nodes. Each node maintains its own transaction snapshot information locally, eliminating the single point of failure while maintaining global transaction visibility through distributed coordination.
Solution Approach 2:
The system transitions from a single-dimensional centralized management model to a multi-dimensional distributed model where transaction management occurs across multiple nodes simultaneously. This dimensional shift enables both high availability and scalability by distributing the management function across the network.
2Loss of information
If a centralized component tracks snapshots globally, then transaction visibility is improved, but scalability is limited
Solution Approach 1:
The snapshot tracking function is extracted from the centralized component and distributed to individual nodes. Each node maintains its own snapshot information, allowing the system to scale horizontally without a central bottleneck while preserving complete transaction visibility through local and remote snapshot access.
Solution Approach 2:
Each distributed transaction manager is designed to be universal, capable of managing both local and remote transactions. This multi-functionality allows any node to handle transaction coordination, enabling the system to scale while maintaining consistent transaction visibility across all nodes.
3Reliability
If distributed transaction management is implemented, then scalability and reliability are improved, but complexity of snapshot coordination increases
Solution Approach 1:
Transaction snapshot information is captured and stored preliminarily at each node before transactions span across multiple nodes. This preliminary action at the source node simplifies coordination by making snapshot data immediately available locally, reducing the complexity of real-time distributed coordination while maintaining reliability.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Embodiments are provided to provide a distributed transaction manager supporting repeatable read isolation level in Massively Parallel Processing (MPP) database systems without a centralized component. Before starting a transaction, a first node identifies a second node involved in the transaction, and requests from the second node a snapshot of current transactions at the second node. After receiving the snapshot from the second node, the first node combines into a reconciled snapshot the snapshot of transactions from the second node with current transactions at the first node. The first node then transmits the reconciled snapshot to the second node and starts the transaction using the reconciled snapshot. A branch transaction is then started at the second node in accordance with the reconciled snapshot. Upon ending the transaction and the branch transaction, the first node and the second node perform a two phase commit (2PC) protocol.