MPP Database Transaction Broker for Real-Time Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Massively parallel processing (MPP) databases face challenges in providing real-time data updates without sacrificing scalability and performance, as they are typically designed for read-only analytics and struggle to handle simultaneous processing of write transactions.
Innovation Solution
The system decouples query processing from memory management by using an MPP execution cluster to process distributed transactions, with a transaction broker managing concurrency, versioning, and timestamps, allowing compute nodes to access and update data asynchronously while maintaining high throughput through distributed query processing and locking mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If MPP database processes bulk inserts during off-peak hours, then data updates can be performed, but real-time data updates cannot be provided without sacrificing scalability and performance
Solution Approach 1:
The system segments the MPP database architecture into separate components: compute nodes for query processing and storage nodes for data management. This segmentation allows write operations to be handled by storage nodes while compute nodes continue parallel processing, enabling real-time updates without sacrificing MPP performance
Solution Approach 2:
A transaction broker is introduced as an intermediary component that manages concurrency control, versioning, and conflict resolution between write transactions and ongoing read operations. This mediator enables real-time data updates while maintaining the scalability and performance of parallel processing through coordinated access management
2Productivity
If MPP database handles simultaneous write transactions, then real-time updates are enabled, but concurrency control and conflict management become complex
Solution Approach 1:
The system implements optimistic concurrency control where transactions operate independently without acquiring locks, and conflicts are detected and resolved automatically through version checking during commit. This self-service approach enables high write throughput while keeping concurrency control mechanisms relatively simple
Solution Approach 2:
The transaction broker implements feedback mechanisms through version timestamps and conflict detection protocols. When write transactions are submitted, the system checks version information to detect conflicts, provides feedback on transaction outcomes, and automatically retries failed transactions, enabling efficient concurrent write handling
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for distributed transaction processing in massively parallel processing (MPP) databases. An embodiment operates by receiving a transaction including a read timestamp corresponding to a version of data of a database associated with the query. It is determined whether a locally stored portion of the data of the database has a local timestamp that is equivalent to or after the read timestamp. A lock may be requested, and if received, the data is updated.