Snapshot-Based File Verification for Data Domain Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file verification methods in Data Domain systems face latency and memory overhead issues during startup, especially in large namespaces, and are prone to correctness errors due to modification timestamp inaccuracies, leading to incomplete verification and unnecessary memory usage.
Innovation Solution
Implementing snapshot-based file verification that uses differential mechanisms to identify and verify only changed files between snapshots, converting data-full snapshots to dataless snapshots for reduced storage and efficient verification, and decoupling snapshot lifecycles between source and replica to manage storage and garbage collection effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If file verification walks through the entire namespace during startup to identify files for verification, then verification completeness is improved, but startup latency increases significantly
Solution Approach 1:
The patent applies preliminary action by maintaining a persistent marker (last verified timestamp) that records the verification state across system restarts. This allows the verification process to resume from where it left off rather than starting over, significantly reducing startup latency while maintaining verification completeness.
Solution Approach 2:
The patent segments the verification process by dividing the namespace into verified and unverified portions based on the persistent marker. Instead of verifying the entire namespace during each startup, only the unverified segment (files modified after the marker timestamp) needs verification, reducing the workload and startup time.
2Reliability
If all files in the namespace are loaded into memory for verification, then verification accuracy is improved, but memory overhead increases significantly
Solution Approach 1:
The patent segments the verification workload by using the persistent marker to identify only the unverified portion of the namespace. This allows the system to load and verify only the necessary files into memory rather than all files, reducing memory overhead while maintaining verification accuracy for the relevant files.
Solution Approach 2:
The patent applies partial action by verifying only the subset of files that are actually needed (those modified after the last verified timestamp) rather than all files in the namespace. This reduces memory usage while ensuring verification accuracy for the critical files.
3Ease of operation
If modification timestamps are used to identify files for verification, then verification process is simplified, but correctness errors occur due to timestamp inaccuracies
Solution Approach 1:
The patent uses modification timestamps as an intermediary mechanism to approximate file change detection. While timestamps alone can be inaccurate, they serve as a useful first-pass filter to identify candidate files for verification, balancing simplicity with reasonable correctness.
Solution Approach 2:
The patent implements feedback by maintaining a persistent marker that records verification state and using it to guide subsequent verification cycles. This feedback loop allows the system to learn from previous verification results and adjust its behavior, improving correctness over time while maintaining operational simplicity.
Data Source
AI summary
In an embodiment, a method can include loading a first snapshot of data stored on a storage device, the first snapshot being verified. The method can further include capturing a second snapshot of data stored on the store device after waiting an interval of time from creation of the first snapshot. The method can further include generating a list of closed files between the two snapshots by differentiating the first snapshot and the second snapshot. The method can additionally include verifying the second snapshot by comparing the closed files in the list of closed files by in the second snapshot to the closed files in the storage device, which is an active snapshot. The method can also include deleting the first snapshot.