Idempotent Journal Mechanism for Distributed File Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
File systems with distributed metadata servers face challenges in maintaining data integrity and coherence during data modification operations, particularly in environments like video editing and archiving, where file names may not be unique over time, leading to potential corruption.
Innovation Solution
Implementing an idempotent journal mechanism with unique 64-bit sequence numbers for each file system object, ensuring that operations are only applied if the sequence numbers match, thereby preventing corruption and allowing for synchronization and restoration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If file names are used to identify file system objects in distributed metadata servers, then operations can be performed and tracked, but file system corruption can occur when file names are not unique over time
Solution Approach 1:
The patent changes the identification parameter from file name to a unique 64-bit sequence number (JSN) that is monotonically increasing and unique for the life of the file system. This parameter change eliminates the ambiguity of non-unique file names while maintaining operational tracking capability.
Solution Approach 2:
The patent introduces an intermediary mechanism (the JSN comparison step) between the journal operation application and the file system object modification. This intermediary verifies uniqueness before applying operations, preventing corruption while allowing the journal mechanism to remain relatively simple.
2Reliability
If journal operations are applied to synchronize data modification operations across distributed metadata servers, then data coherence is improved, but file system corruption can occur due to non-unique file names
Solution Approach 1:
The patent applies preliminary anti-action by checking the JSN before applying journal operations. If the JSN in the journal entry does not match the current JSN of the file system object, the operation is rejected. This preliminary check prevents harmful corruption before it can occur.
Solution Approach 2:
The patent implements feedback by comparing the JSN in the journal entry with the current JSN of the target file system object. This feedback mechanism ensures that only valid operations on the correct objects are applied, maintaining data coherence while preventing corruption.
3Reliability
If unique 64-bit sequence numbers are used for each file system object, then operations can be safely applied and corruption prevented, but the system requires additional storage and processing overhead
Solution Approach 1:
The patent makes the JSN self-service by embedding it directly in the file metadata and using it for both identification and validation purposes. The JSN serves multiple functions (unique identification, operation validation, synchronization) without requiring additional complex validation mechanisms, reducing overall processing overhead.
Data Source
AI summary
Use and maintenance of an idempotent journal in a system having multiple metadata servers. A data modification operation is performed by a metadata server. A journal entry corresponding to the data modification operation is generated. The journal entry includes a file identifier and a journal sequence number corresponding to a data modification operation that created the file.