Metadata Routing Coordinator for Distributed Database Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-partitioned databases, clients face difficulties in determining the responsible metadata node, leading to network congestion and scalability limitations due to excessive network requests for node discovery.
Innovation Solution
A distributed system that maintains strongly consistent metadata copies across multiple nodes using the Raft consensus algorithm, with each partition having a single leader node and client-side caching for efficient request routing, allowing clients to directly communicate with the correct node and fallback mechanisms to ensure high availability and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata is replicated across multiple nodes for high availability, then system reliability is improved, but clients face difficulty determining which node is responsible for the metadata they need, leading to excessive network requests
Solution Approach 1:
The patent introduces a coordinator node as an intermediary that maintains a mapping between metadata keys and responsible metadata nodes. Clients query the coordinator to obtain the correct node for their metadata access needs, eliminating the need for clients to directly determine node responsibility and reducing unnecessary network requests across the distributed system.
2Loss of information
If clients send requests to all metadata nodes to discover the correct node, then the correct node can be located, but network congestion occurs and system scalability is limited
Solution Approach 1:
The patent extracts the node discovery function from the general metadata access process by creating a specialized coordinator node that handles location queries. This separates the routing function from data access, allowing clients to efficiently locate the correct metadata node through the coordinator without broadcasting requests to all nodes, thereby maintaining node discovery accuracy while enabling system scalability.
3Ease of operation
If a centralized approach is used for metadata management, then node routing is simplified, but system flexibility for reconfiguration and dynamic scaling is reduced
Solution Approach 1:
The patent implements dynamic metadata partitioning where the coordinator node maintains flexible mappings that can be reconfigured based on system state. Metadata can be dynamically assigned to different nodes, and the coordinator updates its mapping accordingly. This allows the system to adapt to changing conditions, scale dynamically, and reconfigure partitions without requiring a fixed centralized structure, thereby maintaining routing simplicity while enabling system flexibility.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In some examples, a first computing device may receive, from a second computing device, partition mapping information indicating partitions of a metadata database. The first computing device may be able to communicate with a plurality of metadata nodes, each metadata node maintaining a portion of the metadata database based on the partitioning of the metadata database to distribute the metadata database across the plurality of metadata nodes. The first computing device may determine to send a request to the metadata database based at least on key information. The first computing device may determine, based on the partition mapping information, a first metadata node of the plurality of metadata nodes indicated to maintain a partition of the metadata database corresponding to the key information. The first computing device, may send, to the first metadata node, based on the partition mapping information, a request to perform a database operation.