Restoring Partitioned Database Tables via Parallel Import
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in efficiently backing up and restoring tables that are distributed across multiple machines and partitions, particularly in maintaining consistency and managing resources effectively during backup and restore operations.
Innovation Solution
A method and system for backing up and restoring database tables that involves independent parallel processing of partitions, ensuring consistency within each partition, and managing resources dynamically, including automatic verification of backups and repartitioning to optimize storage and throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If database tables are distributed across multiple machines and partitions, then data availability and security are improved, but backup consistency and operational complexity deteriorate
Solution Approach 1:
The patent segments the backup operation into partition-level independent tasks. Each partition can be backed up independently across different machines, allowing parallel processing while maintaining overall backup consistency. This segmentation resolves the contradiction by enabling distributed backup operations without requiring complex coordination across the entire database system.
Solution Approach 2:
The system performs preliminary actions by establishing backup consistency protocols before actual backup operations begin. Metadata is updated and consistency markers are set in advance, ensuring that when backup operations are executed across distributed partitions, consistency is maintained without requiring complex post-backup verification procedures.
2Reliability
If database tables are distributed across multiple machines and partitions, then data security and disaster recovery are improved, but backup efficiency and resource management deteriorate
Solution Approach 1:
Each partition performs self-service backup operations independently. The system automatically manages backup execution, verification, and completion without requiring centralized coordination for each partition. This self-service approach maintains disaster recovery reliability while significantly improving backup efficiency by eliminating coordination overhead.
Solution Approach 2:
The patent changes the operational parameters from table-level backup to partition-level backup. This parameter change enables parallel execution across multiple partitions, improving backup efficiency while maintaining the reliability benefits of distributed storage through independent partition backup and verification processes.
3Ease of operation
If traditional backup methods are used on distributed databases, then implementation simplicity is maintained, but restore time and data consistency deteriorate
Solution Approach 1:
The restore operation is segmented into independent partition-level restore tasks. Multiple partitions can be restored simultaneously from their respective backups, dramatically reducing total restore time while maintaining simplicity through the same partition-level interface used for backup operations.
Solution Approach 2:
The system maintains continuity of useful action by enabling parallel restore operations across multiple partitions. While traditional methods would restore the entire database sequentially, this approach continuously restores multiple partitions simultaneously, reducing restore time without complicating the operational interface.
4Productivity
If partition-level independent processing is implemented, then backup and restore efficiency are improved, but system complexity and coordination requirements worsen
Solution Approach 1:
Each partition operates as a self-service unit that independently manages its own backup and restore operations. Partition-level processors automatically handle their respective tasks without requiring complex inter-partition coordination, thereby improving efficiency while minimizing system complexity through decentralized autonomous operation.
Data Source
AI summary
A system that implements a data storage service may store data for database tables in multiple replicated partitions on respective storage nodes. In response to a request to restore a given table that was backed up in a remote storage system (e.g., key-value durable storage system), the service may create a new table, and may import a copy of each of the partitions of the given table from the remote storage system into the new table. The request to restore the table may specify a modified value for a configuration parameter for the table or for one of its partitions. The service may apply the new configuration parameter value to the table or its partitions during the restore operation. The new configuration parameter value may indicate an increase or decrease in storage capacity or throughput capacity, and its application may automatically trigger a partition split or move operation.