Distributed Database Backup Partitioning to Cut Metadata Delays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Backing up a large quantity of database files in a distributed system causes delays due to metadata synchronization overhead and bottlenecks in deduplication or consolidation operations, leading to performance issues and file count limits.
Innovation Solution
Aggregating a large set of database files into a single file with partitions, where each partition represents a group of files, and formatting each partition to maintain compatibility across various source file formats and support deduplication and consolidation operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large quantity of database files are backed up using traditional distributed system methods, then complete backup coverage is achieved, but metadata synchronization overhead increases causing delays
Solution Approach 1:
The patent combines multiple individual file metadata synchronization operations into a single aggregated metadata synchronization operation. By consolidating the backup process into one unified operation rather than processing each file separately, the system reduces the cumulative metadata synchronization overhead while maintaining complete backup coverage across all files.
2Loss of substance
If a large quantity of database files are processed through deduplication operations, then data redundancy is reduced, but system bottlenecks increase causing performance degradation
Solution Approach 1:
The patent merges multiple individual file deduplication operations into a single consolidated deduplication process. This approach processes all files together in one operation rather than sequentially, reducing system bottlenecks and improving overall backup processing speed while still achieving effective redundancy reduction.
3Quantity of substance
If a large quantity of database files are consolidated during backup, then storage efficiency is improved, but system limits are reached causing operation failures
Solution Approach 1:
The patent segments the large set of database files into smaller groups or batches during the consolidation process. This segmentation allows the system to process files in manageable chunks that stay within system limits, preventing operation failures while still achieving efficient storage capacity utilization through consolidated backup.
Data Source
AI summary
Methods, systems, and devices for data management are described. A data management system may receive a request to generate backup data for a set of data files from the one or more databases. The data management system may then generate, in response to the request, a file including a set of partitions including respective groups of shard files that correspond to respective groups of data files from among the set of data files. In some examples, a respective group of shard files within a partition of the set of partitions may include a first shard file representative of metadata for the partition and one or more additional shard files representative of the respective group of data files for the partition. The data management system may then distribute the respective groups of shard files to a set of nodes within the distributed backup system.