Database Upgrade via Primary-Secondary Site Switching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Upgrading large databases in online systems, such as multi-tenant systems, often results in significant downtime, which is unacceptable for applications that handle a large number of transactions per second, as conventional techniques require the database to be unavailable for several hours or days during the upgrade process.
Innovation Solution
The system reduces downtime by redirecting client requests from a primary database to a secondary standby database, allowing for the upgrade of the primary database while maintaining availability, by using a combination of logical and physical replication to synchronize data across databases and switch processing responsibilities between sites.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the database is upgraded using conventional techniques, then the database version is updated to support new features and security fixes, but the database becomes unavailable for several hours or days causing significant downtime
Solution Approach 1:
The database system is segmented into multiple independent database instances (first database instance, second database instance, third database instance) that can operate independently. This allows the upgrade process to be divided into separate phases where different instances are upgraded at different times, ensuring continuous availability of at least one instance throughout the upgrade process.
Solution Approach 2:
A fourth database instance is prepared in advance with the latest database version before the upgrade process begins. This pre-prepared instance serves as a ready-to-switch backup that can immediately take over if issues arise during the upgrade of active instances, minimizing potential downtime and ensuring business continuity.
Solution Approach 3:
The system introduces intermediary components including a replication manager that coordinates data synchronization between instances and a switch manager that handles the redirection of client requests. These intermediaries enable seamless transitions between database instances during upgrades, maintaining availability without requiring direct manual intervention.
2Reliability
If the database remains available during upgrade by using standby databases, then availability is maintained, but the system complexity increases with multiple database instances and replication mechanisms
Solution Approach 1:
The replication manager implements self-service automation by automatically detecting which database instances require upgrading, coordinating the upgrade sequence, and managing data replication between instances without requiring manual intervention. This automation reduces the operational complexity of managing multiple database instances despite the increased architectural complexity.
Solution Approach 2:
The switch manager monitors the health and status of database instances continuously and provides feedback to automatically determine when to switch between instances. This feedback mechanism enables the system to self-regulate and maintain availability dynamically without complex manual control, offsetting the architectural complexity through intelligent automation.
Data Source
AI summary
An online system performs upgrades of replicated databases. The online system stores data in a primary site and a secondary site. The primary site comprising databases used to process client requests and a secondary site comprising databases used as standby. The online system upgrades databases of both sites by upgrading the secondary site, switching client requests to the secondary site while using the databases of primary site as standby databases. The online system then upgrades the primary site. The process allows the online system to upgrade databases with small down time.