Replicated Database Upgrade via Traffic Quiescing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database management system upgrades require shutdowns, disrupting service and affecting multiple users in multi-tenant systems, as they cannot be performed while maintaining high availability.
Innovation Solution
A replicated database management system with a master node, read-replica nodes, and a spare node is used, where upgrades are performed in a specific order, and traffic is quiesced and redirected to ensure continuous availability, employing an immutable infrastructure approach by installing new instances and moving traffic to them.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional upgrade techniques are used, then the database system can be upgraded, but service disruption occurs and availability decreases
Solution Approach 1:
The database system is segmented into multiple nodes (primary database node, secondary database nodes, read-replica nodes) that can operate independently. This allows the system to perform upgrades on specific nodes without shutting down the entire database system, thereby maintaining service availability during the upgrade process.
Solution Approach 2:
The system performs preliminary actions by creating a new version of the database management system in advance on secondary nodes or read-replica nodes before the actual upgrade. This allows the upgrade to be prepared and tested beforehand, enabling seamless switching to the new version without service disruption.
2Ease of manufacture
If the database system is shut down for upgrade, then the upgrade can be performed, but service disruption affects multiple users and enterprises
Solution Approach 1:
The system introduces intermediary components including a quiescer that coordinates traffic redirection and a reader that assists in reading data during the upgrade process. These intermediaries enable the upgrade to proceed without directly disrupting service, as traffic is smoothly redirected through these mediator components.
Solution Approach 2:
The system creates a copy of the database management system on secondary nodes and read-replica nodes. This copy allows the upgrade to be performed on the replica rather than the primary system, enabling the upgrade process to proceed independently of active service operations.
3Reliability
If rolling upgrade is performed on replicated database nodes, then availability is maintained, but system complexity increases
Solution Approach 1:
The system implements self-service mechanisms where the quiescer automatically coordinates traffic redirection, and the reader autonomously handles data reading operations during the upgrade process. This automation reduces the need for manual intervention and simplifies the complex rolling upgrade procedure across multiple replicated nodes.
Data Source
AI summary
An online system, such as a multi-tenant system ensures high availability of systems, for example, database management systems. The online system replicates the databases across multiple datacenters including: (1) a master node that receives read and write requests (2) a read-replica that receives only read requests and (3) a spare node that does not receive requests but acts as standby for high availability. One or more application servers may send read and write requests to the databases. The system performs a sweep of upgrades of the database nodes and also performs traffic quiescing of the requests received from the application servers to redirect the traffic across the database nodes as the upgrade sweep is orchestrated. The sweep of upgrades ensures that the availability of the database management system to the end users is maximized during the upgrade process.