Fileset Partitioning for Parallel Snapshot Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing complexity of data management and storage, particularly in virtualized environments, necessitates improved approaches for data backup and recovery, especially in cases of failure, as existing methods struggle with scalability and efficiency in handling large volumes of data.
Innovation Solution
A data management and storage (DMS) cluster divides filesets into partitions and uses peer DMS nodes to autonomously execute snapshot jobs in parallel, leveraging a distributed data store across nodes to optimize performance and fault tolerance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data backup is performed on large volumes of data in virtualized environments, then data recovery reliability is improved, but I/O bottlenecks increase and backup efficiency deteriorates
Solution Approach 1:
The patent divides a fileset into multiple partitions, allowing backup operations to be performed on each partition independently and in parallel. This segmentation enables multiple DMS nodes to simultaneously process different partitions, thereby improving backup efficiency while maintaining data recovery reliability through distributed operations.
Solution Approach 2:
The patent transitions from a single-node sequential backup approach to a multi-node parallel backup approach by distributing partitions across multiple DMS nodes. This dimensional change from one-dimensional sequential processing to multi-dimensional parallel processing resolves the contradiction between reliability and efficiency.
2Device complexity
If snapshot operations are centralized on single nodes, then system complexity is reduced, but processing speed and fault tolerance deteriorate
Solution Approach 1:
The patent segments snapshot operations by assigning different partitions to different DMS nodes, enabling parallel execution of snapshot jobs. This segmentation increases processing speed while distributing the operational complexity across multiple nodes rather than concentrating it on a single node.
Solution Approach 2:
The patent combines multiple DMS nodes into a collaborative backup system where each node performs snapshot operations on assigned partitions. This merging of multiple processing units achieves higher aggregate processing speed while maintaining manageable complexity through standardized node operations.
3Device complexity
If all snapshot jobs are executed by single DMS nodes, then coordination overhead is reduced, but fault tolerance and recovery capability deteriorate
Solution Approach 1:
The patent segments the backup workload across multiple DMS nodes, with each node responsible for specific partitions. This segmentation inherently improves fault tolerance because failure of one node does not affect others, while coordination overhead remains manageable through standardized partition assignment and autonomous node operation.
Solution Approach 2:
Each DMS node autonomously executes snapshot jobs for its assigned partitions without requiring constant coordination with other nodes. This self-service approach maintains low coordination overhead while achieving high fault tolerance through distributed independent operation of multiple nodes.
Data Source
AI summary
In one approach, filesets to be backed up are divided into partitions and snapshots are pulled for each partition. In one architecture, a data management and storage (DMS) cluster includes a plurality of peer DMS nodes and a distributed data store implemented across the peer DMS nodes. One of the peer DMS nodes receives fileset metadata for the fileset and defines a plurality of partitions for the fileset based on the fileset metadata. The peer DMS nodes operate autonomously to execute jobs to pull snapshots for each of the partitions and to store the snapshots of the partitions in the distributed data store.