Proxy-Based Transaction Request Flow for GTM Concurrency Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The centralized Global Transaction Manager (GTM) in multi-master distributed database systems becomes a bottleneck due to high concurrency, leading to connection limitations and inefficient processing of concurrent transactions, especially in large-scale deployments.
Innovation Solution
Implementing a proxy-based method with a proxy module in each master node to manage data processing requests, allowing dynamic connection choices and using shared memory for efficient communication, thereby reducing network traffic and workload on the GTM.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a centralized Global Transaction Manager (GTM) is used to support snapshot isolation in a multi-master MPP architecture, then transaction consistency and isolation can be maintained, but the GTM assembly becomes a bottleneck under high concurrency, causing connection limitations and increased system pressure
Solution Approach 1:
The patent segments the centralized GTM functionality by introducing local GTM components at each master node. Instead of a single centralized GTM handling all transactions, each master node has its own GTM instance that can independently manage transactions locally. This segmentation distributes the transaction management workload across multiple nodes, eliminating the single-point bottleneck while maintaining transaction consistency through coordinated communication between local GTM instances.
Solution Approach 2:
The patent introduces a proxy module as an intermediary between clients and the GTM assembly. The proxy module receives client requests, performs preliminary processing and validation, and only forwards necessary requests to the GTM assembly. This intermediary layer reduces the direct connection burden on the GTM, filters out unnecessary requests, and manages connection pooling, thereby improving the system's concurrency handling capability while maintaining transaction integrity.
2Productivity
If multiple processes or threads are used to improve GTM concurrency, then processing capacity increases, but the centralized nature of the GTM assembly remains, and connection pressure and operational complexity continue to grow
Solution Approach 1:
The patent divides the GTM functionality into multiple independent local GTM instances distributed across different master nodes. Each local GTM instance operates semi-independently, managing its own transaction context and state. This segmentation transforms the single centralized GTM into a distributed collection of GTM instances, increasing processing capacity through parallel operation while reducing operational complexity by localizing management responsibilities at each node.
Solution Approach 2:
Each local GTM instance at a master node is designed to be self-sufficient, handling its own transaction management, connection pooling, and state maintenance without requiring direct intervention from the central GTM assembly for every operation. The local GTM instances can independently process transactions within their jurisdiction, reducing the operational burden and complexity on the centralized system while maintaining overall system coordination through standardized protocols.
Data Source
AI summary
The present application provides a data processing method, a system, a computing device and a computer readable storage medium, wherein the data processing method is applied to a data processing node of a distributed data processing system and includes: receiving a plurality of data processing requests sent by a client, determining a target processing amount based on the number of the plurality of data processing requests, and limiting a flow of the plurality of data processing requests in accordance with the target processing amount to obtain a target data processing request; forwarding the target data processing request to a global transaction manager, receiving a processing result from processing the target data processing request by the global transaction manager, and returning the processing result to the client corresponding to each data processing request.