Epoch-Based Snapshot Chunk Retention in Distributed File Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed file systems, especially in cloud-based object stores, file segments or 'chunks' may be deleted prematurely due to lack of references, even if they are still associated with snapshots or backups, leading to data loss and inefficiencies in management.
Innovation Solution
Implementing an epoch-based management system that assigns monotonically increasing identifiers to file segments, using an epoch table to track segments created before or after snapshots, and a 'death' table to determine when segments are no longer referenced, ensuring retention of chunks associated with still-stored snapshots.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If reference count is used to manage file segments, then deletion efficiency is improved, but data reliability deteriorates due to premature deletion of segments still needed by snapshots
Solution Approach 1:
The patent segments the reference counting mechanism into two independent components: a reference count for live file system objects and a separate snapshot reference mechanism. This allows the system to track and manage segments referenced by snapshots independently from those referenced by live files, preventing premature deletion while maintaining efficient deletion of truly unreferenced segments.
Solution Approach 2:
The patent introduces an intermediary mechanism (snapshot reference tracking) that mediates between the reference count system and the segment deletion process. This intermediary layer ensures that segments are not deleted based solely on reference count, but only after verifying they are not needed by any retained snapshots, thus resolving the conflict between deletion efficiency and data integrity.
2Reliability
If segments are retained for snapshot compatibility, then data reliability is improved, but storage resource utilization deteriorates due to retention of unnecessary segments
Solution Approach 1:
The patent changes the parameter used for deletion decisions from a simple reference count to a compound parameter that considers both reference count and snapshot retention status. This parameter change enables the system to make more accurate decisions about which segments to retain and which can be safely deleted, optimizing storage resource utilization while maintaining snapshot compatibility.
Solution Approach 2:
The patent implements a mechanism to discard (delete) segments that are no longer needed by either live files or snapshots, while recovering (retaining) only those segments that are still referenced. This selective discarding and recovering process ensures that storage resources are not wasted on unnecessary segments while maintaining the reliability needed for snapshot compatibility.
Data Source
AI summary
Epoch-based management of file system data is disclosed. In various embodiments, for each of a plurality of snapshots, data that associates with the snapshot a corresponding last chunk identifier assigned as of a time at which the snapshot is or was created is stored a in a first data structure. For each of a plurality of chunks of file system data no longer referenced by any live file system object, an epoch identifier associated with an epoch in which the chunk became no longer referenced by any live file system object is stored in a second data structure. A chunk identifier and associated epoch comprising an entry in the second data structure are compared with one or more entries in the first data structure to determine whether a chunk associated with the chunk identifier is associated with a snapshot that continues to be retained.