Parallel Batch Metadata Transfer in Sharded Columnar Database
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In columnar databases, updating out-of-date metadata objects across multiple nodes can significantly degrade database performance due to the need for serial transactions and locks, leading to prolonged downtime and decreased efficiency during high usage periods.
Innovation Solution
A coordinator node identifies source and target node pairs within a cluster to perform a parallel batch metadata transfer update, updating multiple nodes in a single transaction, thereby reducing the time required to synchronize metadata and minimizing lock duration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If serial transactions with locks are used to update metadata objects across multiple nodes, then data consistency is maintained, but database performance degrades significantly and downtime is prolonged
Solution Approach 1:
The system segments the metadata update process into individual node updates that can be performed independently. Each node receives and processes metadata updates separately through its own transaction, eliminating the need for a single serial transaction that locks all nodes. This segmentation allows concurrent updates across multiple nodes while maintaining data consistency within each node's transaction context.
Solution Approach 2:
The system introduces an intermediary mechanism where metadata updates are transmitted to a coordinator node first, which then distributes them to target nodes. This intermediary approach allows the system to coordinate updates without requiring all nodes to be locked simultaneously, enabling parallel processing while maintaining consistency through the intermediary's control.
2Manufacturing precision
If serial transactions with locks are used to update metadata objects, then update accuracy is ensured, but the time required for synchronization increases
Solution Approach 1:
The update process is divided into independent node-level transactions that can execute simultaneously. Each target node processes its own metadata update independently with its own lock, allowing parallel execution across multiple nodes. This segmentation maintains update accuracy through individual transaction integrity while dramatically reducing total synchronization time through concurrent processing.
Solution Approach 2:
The system performs preliminary actions by first transmitting metadata updates to the coordinator node before initiating parallel updates at target nodes. This preliminary coordination allows the system to prepare update data in advance, then execute synchronized updates across multiple nodes simultaneously, reducing overall synchronization time while maintaining accuracy through pre-coordinated update content.
3Productivity
If parallel batch metadata transfer update is implemented, then database performance is enhanced and downtime is reduced, but system complexity increases
Solution Approach 1:
The system uses a coordinator node as an intermediary to manage the complexity of parallel updates. The coordinator receives metadata updates, identifies source and target node pairs, and coordinates the parallel execution of updates across multiple nodes. This intermediary approach centralizes the coordination logic, allowing parallel processing to proceed without each node needing to independently manage complex synchronization protocols, thus enhancing performance while controlling system complexity.
Data Source
AI summary
A database management system includes a cluster of nodes over which a columnar database has been sharded into shards to which the nodes are subscribed. Source nodes of the cluster store up-to-date metadata objects for the shards, and target nodes of the cluster storing out-of-date metadata objects for the shards. A coordinator node initiates a parallel batch metadata transfer update process in which the out-of-date metadata objects at the target nodes are updated in parallel from the up-to-date metadata objects at the source nodes. The coordinator node can determine pairs of source nodes and target nodes; the source node and the target node of each pair respectively store an up-to-date metadata object and an out-of-date metadata object for a same shard.