Sharded OLTP Transaction Routing to Reduce GTM Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed online transaction processing (OLTP) systems with sharded databases face performance degradation due to synchronization overhead and bottlenecks caused by centralized global transaction managers (GTM) when handling multi-node transactions.
Innovation Solution
Implement a GTM-lite approach that distinguishes between single-shard and multi-shard transactions, optimizing single-shard transactions by avoiding GTM interactions and managing multi-shard transactions through a centralized GTM only, reducing unnecessary communication and synchronization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a centralized global transaction manager (GTM) is used to synchronize multi-shard transactions, then global consistency is improved, but system throughput and scalability deteriorate due to synchronization overhead and bottlenecks
Solution Approach 1:
The patent segments transaction management into two categories: single-shard transactions handled locally by data nodes without GTM involvement, and multi-shard transactions requiring GTM coordination. This segmentation allows the majority of transactions to bypass the GTM bottleneck, improving overall throughput while maintaining consistency for distributed transactions through selective application of centralized management.
Solution Approach 2:
The patent applies different transaction management strategies based on the specific characteristics of each transaction type. Single-shard transactions receive local, autonomous handling with direct data node processing, while multi-shard transactions receive centralized coordination. This local quality approach optimizes performance for each transaction category according to its specific requirements.
2Reliability
If GTM interactions are performed for all transactions, then global consistent view is maintained, but latency increases due to unnecessary communication overhead
Solution Approach 1:
The patent extracts GTM interactions from the common transaction path, removing them as a mandatory step for all transactions. Instead, GTM involvement is extracted and applied only to multi-shard transactions that genuinely require centralized coordination for consistency, thereby eliminating unnecessary communication overhead for single-shard transactions and reducing overall latency.
3Productivity
If data is distributed across multiple shards to achieve scalable performance, then processing capacity is improved, but synchronization complexity increases for multi-node transactions
Solution Approach 1:
The patent segments the synchronization problem by identifying and separating single-shard transactions from multi-shard transactions. This segmentation simplifies the synchronization complexity by applying lightweight local management to single-shard operations and reserved centralized coordination only when multi-node synchronization is genuinely required, thereby managing complexity proportionally to actual needs.
Data Source
Figure 1
Figure 2A~2B
Figure 3
AI summary
A system and method for improved online transaction processing (OLTP) in a sharded database is provided. Overhead associated with a global transaction manager is reduced and scalability improved by determining whether incoming queries are single-shard transactions or multi-shard transactions. For multi-shard transactions, a distributed transaction ID (DXID) is requested from the GTM, and then forwarded with the query to one or more data notes. For single-shard transactions, the query is sent to a data node without requesting a DXID from the GTM.