Cloud Database Shard Recovery via Parallel Node Restoration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face challenges in scalability, cost-effectiveness, and recovery efficiency, particularly for small tenants and in scenarios where storage and compute layers are disaggregated.
Innovation Solution
The system logically splits each tenant's data into shards, distributing them across multiple storage nodes for scalability and redundancy. Each shard is backed up to an object storage service, allowing for rapid recovery and incremental data access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all storage nodes are recovered simultaneously using traditional backup methods, then data redundancy is restored, but recovery time and network resource consumption increase significantly
Solution Approach 1:
The patent divides the storage layer into multiple independent storage nodes, each capable of autonomous recovery. When a failure occurs, only the affected storage node(s) are recovered individually rather than recovering the entire storage layer simultaneously. This segmentation enables parallel recovery operations and reduces the overall recovery time while maintaining data redundancy through selective node restoration.
Solution Approach 2:
The system pre-positions backup data in cloud object storage before failures occur. When a storage node fails, the recovery process simply needs to retrieve and restore data from the already-prepared cloud backups rather than performing complex data reconstruction. This preliminary preparation of backup data significantly accelerates the recovery process while ensuring data redundancy is restored.
2Speed
If networking resources are overprovisioned to handle data-intensive recovery operations, then recovery speed improves, but resource allocation efficiency deteriorates during normal operations
Solution Approach 1:
The patent implements dynamic resource allocation where networking resources are provisioned based on actual operational needs rather than peak recovery requirements. During normal operations, resources are allocated efficiently for routine database operations. During recovery operations, the system dynamically scales network resources to handle data-intensive restore operations from cloud object storage, then releases these resources afterward. This dynamic approach maintains high recovery speed while improving overall resource allocation efficiency.
3Adaptability or versatility
If storage and compute layers are disaggregated for independent scaling, then scalability improves, but system complexity increases
Solution Approach 1:
The patent creates a universal storage node architecture that can function independently whether part of a disaggregated storage layer or integrated with compute resources. Each storage node is self-contained with its own data volumes, log volumes, and recovery capabilities, allowing it to be scaled independently while maintaining simplicity. The standardized node design reduces architectural complexity despite the disaggregated structure, as nodes can be deployed and managed uniformly across different configurations.
Data Source
AI summary
Systems and methods include storage a backup of a first shard of a first database table of a database in a first backup location, the first shard including a first key range of the first database table, storage of a backup of a second shard of the first database table in a second backup location, the second shard including a second key range of the first database table, reception of an instruction to recover the database, and, in response to the instruction, recovery of the first shard to a first storage node from the first backup location and, in parallel, recover the second shard to a second storage node from the second backup location.