Compliance Namespace Separation for Snapshot Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed file systems face challenges in maintaining data retention compliance during failover and failback scenarios, as incremental sync processes can leave files in an inconsistent state, particularly when the sync job is interrupted, leading to potential data loss and non-compliance with regulatory requirements.
Innovation Solution
The solution involves establishing a new sync between a source and target cluster, where a target resident source compliance store is generated, and canonical links are created to ensure files are committed and retained correctly, even if the sync is interrupted, by using a system b-tree to track files for WORM state compliance and applying commits only after the sync is complete.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If incremental sync process applies WORM state commits to each file after sync finishes, then data retention compliance is maintained, but files in inconsistent state may be committed to WORM state on target cluster when sync fails or is cancelled
Solution Approach 1:
The patent creates a compliance store on the target cluster before the incremental sync completes. This preliminary action ensures that the target cluster has the necessary compliance infrastructure ready, and files can be properly tracked and committed to WORM state even if the sync process is interrupted. The compliance store acts as a buffer that maintains compliance state independently of the sync process completion status.
Solution Approach 2:
The compliance store serves as an intermediary between the incremental sync process and the WORM state commitment. It tracks which files should be committed to WORM state based on source cluster snapshots, and manages the commitment process separately from the data sync. This intermediary layer ensures that only files that have been properly synchronized and marked for compliance are committed, preventing inconsistent file states from being wrongly committed.
2Reliability
If files are individually synced from source cluster to target cluster with WORM state commits, then data is transferred with retention properties, but sync process complexity increases and interruption points create compliance risks
Solution Approach 1:
The patent segments the compliance management into a separate compliance store component that operates independently from the data sync process. The compliance store manages WORM state commitments for individual files, while the sync process handles data transfer. This segmentation allows each process to be optimized independently and reduces the complexity of coordinating compliance state with data synchronization.
Solution Approach 2:
The patent creates a copy of the source cluster's compliance metadata in the target cluster's compliance store. This copy contains the snapshot information and file mapping data needed to determine which files should be committed to WORM state. By copying the compliance metadata separately from the data, the system can maintain retention properties without requiring complex coordination between sync and compliance operations.
3Reliability
If canonical links are sent during sync process, then all files are properly tracked for compliance, but sync time increases and interruption may leave files uncommitted
Solution Approach 1:
The compliance store is created and populated with snapshot information from the source cluster before the incremental sync begins. This preliminary action establishes the compliance tracking framework in advance, so that when files are synchronized, their compliance status can be determined quickly without requiring time-consuming compliance checks during the sync process itself.
Solution Approach 2:
The compliance store maintains continuous tracking of file compliance status throughout the sync process. By keeping the compliance metadata updated and accessible during sync operations, the system can quickly determine which files need WORM state commitment without interrupting the sync process for compliance verification. This continuous action ensures compliance tracking accuracy while minimizing additional sync time.
Data Source
AI summary
Embodiments of the present disclosure relate to creating and maintaining a separate compliance store within a domain. Files that are committed within the domain are linked to the compliance store using canonical links. Canonical links are guaranteed to be non-erasable and non-rewritable for the duration of their retention period. When using a source cluster of nodes as a primary production cluster and a target cluster of nodes as a backup cluster, file data and canonical links can be synced from the source cluster to the target cluster to maintain compliance with data retention rules. Non-canonical links can be modified as necessary to ensure snapshot consistency when failover or failback occurs between a source cluster of nodes and a target cluster of nodes.