Distributed Filesystem Checkpointing via Global Data-less Snapshots
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed filesystems face challenges in efficiently taking synchronized checkpoints across multiple nodes, leading to significant data loss and impractical rollback processes due to the large amount of data generated and downtime incurred.
Innovation Solution
The implementation of a global snapshot process that captures a single point in time for spanning file sets, allowing for efficient checkpointing and recovery by generating global data-less snapshots across meta and data file set objects, with incremental distributed recovery and snapshot event logging to support per-file set basis restoration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional checkpointing methods are used across multiple nodes, then data loss can be mitigated, but the amount of data generated is large and rollback processes become impractical
Solution Approach 1:
The patent segments the distributed filesystem into multiple independent spanning file sets, each with its own metadata and data objects. Checkpoints are taken independently for each spanning file set rather than for the entire filesystem, reducing the total data volume that needs to be checkpointed while maintaining reliability through distributed checkpointing across segments.
2Reliability
If traditional checkpointing is performed across multiple nodes, then data recovery is possible, but significant downtime is incurred
Solution Approach 1:
The patent implements preliminary actions by continuously maintaining metadata checkpoints and using asynchronous data checkpointing in the background. When a failure occurs, the system can immediately roll back using pre-prepared metadata checkpoints while data is recovered asynchronously, minimizing downtime and allowing the filesystem to become operational faster.
3Stability of the object's composition
If synchronized checkpoints are taken across all nodes, then consistency is achieved, but the complexity of coordination increases
Solution Approach 1:
The patent divides the filesystem into independent spanning file sets that can be checkpointed separately. This segmentation reduces coordination complexity because each spanning file set can be checkpointed independently without requiring synchronization across all nodes for the entire filesystem, while still maintaining overall consistency through the distributed architecture.
4Reliability
If full data checkpoints are taken, then complete recovery state is captured, but resource impact and storage overhead increase
Solution Approach 1:
The patent extracts and separates metadata from data objects, maintaining complete checkpoints for metadata while using incremental or asynchronous checkpointing for data. This extraction allows the system to capture the essential recovery state in metadata checkpoints without the heavy resource cost of checkpointing all data simultaneously, reducing storage overhead and resource impact.
Data Source
AI summary
Disclosed herein are techniques for generating fractional checkpoints on a distributed file system by snapshotting subcomponents referred to as “file sets.” The techniques include capturing a present time; calculating from configured parameters a future wakeup time for a subsequent set of checkpoints from the present time; walking a database of meta file set objects to locate at least one meta file set object; calculating a retention period for a snapshot that is being created; and generating a global data-less snapshot for the meta file set object and remote data file set objects associated with the meta file set object, and then repeating the process for subsequent file set objects in the database.