Partitioned Database Backup via Independent Parallel Upload
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Efficiently backing up and restoring database tables distributed across multiple machines is challenging due to the complexity of maintaining consistency across partitions, especially in large databases with distributed data.
Innovation Solution
A method for backing up and restoring database tables that involves independently backing up each partition in a scalable data storage system, allowing for asynchronous operations while ensuring consistency within each partition, and supporting flexible configuration changes during the restore process, including changes to storage resources and indexing options.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If database tables are distributed across multiple machines for scalability and availability, then system reliability and data availability improve, but backup complexity and consistency maintenance difficulty increase
Solution Approach 1:
The patent divides the database table into multiple independent partitions that are distributed across different machines. Each partition can be backed up independently, which simplifies the backup process while maintaining the benefits of distribution. The partitioning strategy allows parallel backup operations and reduces consistency maintenance complexity.
Solution Approach 2:
The system performs preliminary actions by capturing the state of each partition before the actual backup operation. This includes recording metadata about partitions, their locations, and their states, which enables consistent backup reconstruction without requiring complex real-time coordination across distributed machines.
2Productivity
If database tables are partitioned into multiple independent parts for scalability, then system performance and data locality improve, but backup consistency and operation complexity worsen
Solution Approach 1:
The database is segmented into independent partitions that can be processed in parallel, maintaining high system performance for read operations. The backup system leverages this segmentation by operating on each partition independently, which actually simplifies the backup operation compared to treating the entire distributed table as a single unit.
Solution Approach 2:
The patent uses copying mechanisms to replicate partition metadata and state information to backup storage. This copying approach preserves the distributed structure and performance benefits while creating a consistent backup copy that can be restored independently, simplifying the overall backup operation.
3Reliability
If data is replicated across multiple machines for disaster recovery, then system reliability improves, but backup time and data transfer requirements increase
Solution Approach 1:
By segmenting the replicated data into distinct partitions, the backup system can process and transfer data in parallel across multiple partitions simultaneously. This reduces the total backup time while maintaining the replicated structure needed for disaster recovery. Each partition's backup can proceed independently without waiting for other partitions.
Solution Approach 2:
The backup system maintains continuous operation by performing backup activities during low-traffic periods or in the background, without interrupting normal database operations. The incremental backup mechanism allows continuous updates to backup storage, reducing the time required for full backup operations while maintaining disaster recovery capability.
Data Source
AI summary
A system that implements a data storage service may store data for a database table in multiple replicated partitions on respective storage nodes. In response to a request to back up a table, the service may back up individual partitions of the table to a remote storage system independently and (in some cases) in parallel, and may update (or create) and store metadata about the table and its partitions on storage nodes of the data storage service and/or in the remote storage system. Backing up each partition may include exporting it from the database in which the table is stored, packaging and compressing the exported partition for upload, and uploading the exported, packaged, and compressed partition to the remote storage system. The remote storage system may be a key-value durable storage system in which each backed-up partition is accessible using its partition identifier as the key.