Contingency Resource Management in Multi-AZ Database Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database systems face challenges in scaling and maintaining high availability and durability while minimizing costs, particularly due to the difficulty in distributing databases while ensuring ACID properties and the inefficiency of redundant resource provisioning for fault tolerance.
Innovation Solution
The system implements a distributed database architecture with a separate storage layer that offloads backup, restore, and recovery operations, using a distributed storage system to handle redo log records and perform crash recovery lazily, allowing the database engine to restart quickly and maintain high availability without waiting for full data restoration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If redundant resource instances are provisioned to provide durability and availability, then reliability is improved, but device complexity and cost increase
Solution Approach 1:
The system segments resources into primary resource instances and contingency resource instances across different availability zones. Primary instances handle normal operations while contingency instances stand by for failover, dividing the redundancy function into distinct roles that reduce overall system complexity while maintaining reliability.
Solution Approach 2:
Contingency resource instances are provisioned in advance and kept in a standby state before failures occur. This preliminary provisioning ensures that backup resources are immediately available when needed, eliminating the need to provision and configure redundant resources after a failure event, thus reducing complexity while ensuring reliability.
2Reliability
If crash recovery includes restoring from backup and replaying changes, then reliability is improved, but loss of time increases
Solution Approach 1:
The system performs preliminary actions by maintaining contingency resource instances in a pre-provisioned standby state with synchronized data. When a failure occurs, these pre-prepared instances can immediately take over, eliminating the time-consuming processes of restoring from backup and replaying changes, thus reducing recovery time while maintaining reliability.
Solution Approach 2:
The system creates and maintains copy instances that replicate data and state from primary instances in real-time or near real-time. These copies serve as ready-to-use backups that can immediately assume operations upon failure, dramatically reducing recovery time compared to traditional restore-and-replay methods while ensuring data reliability.
3Reliability
If resources are reserved for long duration, then reliability is improved, but loss of time and flexibility worsen
Solution Approach 1:
The system implements dynamic resource allocation where contingency resource instances can be provisioned, maintained in standby, activated upon failure, and then re-provisioned or released based on ongoing needs. This dynamic approach allows the system to adapt resource allocation to changing conditions while maintaining reliability through automated failover capabilities.
Solution Approach 2:
The system enables temporary reservation of contingency resources that can be activated, used for a specific purpose (such as handling a failure or surge), and then released or re-provisioned when no longer needed. This allows flexible resource management where backup capacity is maintained only when necessary, improving adaptability while preserving reliability through on-demand activation.
Data Source
AI summary
A network-based services provider may reserve and provision primary resource instance capacity for a given service (e.g., enough compute instances, storage instances, or other virtual resource instances to implement the service) in one or more availability zones, and may designate contingency resource instance capacity for the service in another availability zone (without provisioning or reserving the contingency instances for the exclusive use of the service). For example, the service provider may provision resource instance(s) for a database engine head node in one availability zone and designate resource instance capacity for another database engine head node in another availability zone without instantiating the other database engine head node. While the service operates as expected using the primary resource instance capacity, the contingency resource capacity may be leased to other entities on a spot market. Leases for contingency instance capacity may be revoked when needed for the given service (e.g., during failover).