Sharded Database Cluster Consistency via Load Balancer Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database management systems face performance bottlenecks and troubleshooting difficulties due to the partitioning of databases across multiple servers, particularly in handling large volumes of data and high request rates, where master database servers can become bottlenecks and identifying faulty servers is challenging.
Innovation Solution
Implementing a sharded database system with a load balancer that assigns queries to servers within the same server set, allowing each server to process queries for its own shard and simplifying troubleshooting by ensuring requests go only to other servers within the same set, thereby reducing unnecessary routing and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a database is partitioned across multiple servers to handle large data volume and high request rates, then the system capacity and throughput are improved, but performance bottlenecks occur at master database servers and troubleshooting becomes difficult
Solution Approach 1:
The database system is segmented into multiple independent server sets, where each server set contains multiple servers that collectively handle a portion of the database. This segmentation allows the system to scale horizontally while maintaining manageable complexity within each server set, resolving the contradiction between handling large data volumes and maintaining system simplicity.
2Reliability
If master database servers are used to coordinate updates across partitioned databases, then data consistency is maintained, but the master servers become performance bottlenecks
Solution Approach 1:
The master database server coordination function is extracted and replaced with a distributed consensus mechanism where servers within each server set collectively manage updates. This eliminates the single point of contention at master servers, allowing parallel processing while maintaining consistency through the checkpoint roll call mechanism, thus resolving the bottleneck issue.
3Adaptability or versatility
If queries are routed across all servers in a partitioned database, then comprehensive data access is enabled, but unnecessary routing increases response time
Solution Approach 1:
Each server set is configured to handle queries locally for its portion of the database, eliminating the need for queries to route through all servers. This local化处理 enables comprehensive data access across the distributed system while significantly reducing query response time by avoiding unnecessary cross-server routing, resolving the contradiction between access capability and response time.
4Quantity of substance
If traditional database partitioning is implemented across multiple servers, then data storage capacity is increased, but identifying faulty servers becomes challenging
Solution Approach 1:
The system is divided into distinct server sets with clear boundaries and responsibilities. Each server set independently manages its portion of the database, which isolates faults to specific server sets. This segmentation makes fault identification straightforward by allowing administrators to pinpoint problematic areas without having to investigate the entire distributed system, resolving the contradiction between storage capacity and fault detectability.
Data Source
AI summary
A system and method for maintaining consistency across a decentralized database cluster is presented. A database cluster can contain a plurality of database servers, each hosting a copy of the same database. In a system without a centralized coordinating server, a message queue can be used to simultaneously provide updates to each database server in the database cluster. A When a database server receives an index update message, it adds message to a queue indicating it has received the message and watches an index update node. Once every server in the cluster has added a message to the queue, it is known that every database server has received the message. The update can be performed for each server and the changes committed simultaneously. In such a manner, all database servers in the cluster perform an update at approximately the same time. Other embodiments also are disclosed.