Directory Level Atomic Commit Protocol for Distributed Storage Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed filesystems and storage systems often produce inconsistent read results after modifications, leading to issues with race conditions and incomplete data when used in cluster computing environments, particularly when reading intermediate results from partially completed processes.
Innovation Solution
Implementing a directory level atomic commit system that uses unique transaction identifiers, start and committed marker records, and an external transaction manager to ensure strong atomicity and snapshot isolation, allowing for consistent listings and garbage collection to manage uncommitted files, thereby preventing anomalies and ensuring data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If distributed storage is used to store intermediate results from cluster computing jobs, then storage capacity and accessibility are improved, but data consistency and reliability deteriorate due to race conditions and inconsistent read results from partially completed processes
Solution Approach 1:
The system performs preliminary actions by creating marker records (start markers and committed markers) before and after writing data files. These markers are placed in advance to indicate the transaction state, allowing readers to determine whether to include or exclude files based on the commit status. This preliminary marking mechanism ensures that readers can consistently identify committed transactions without encountering race conditions.
Solution Approach 2:
Marker records serve as intermediaries between writers and readers in the distributed storage system. The start marker and committed marker act as mediators that convey transaction state information without requiring direct communication between writers and readers. Readers query these marker records to determine the commit status of transactions, eliminating the need for complex coordination and ensuring consistency without sacrificing storage versatility.
2Loss of information
If directory listings are provided to reflect all files, then completeness of information is improved, but data integrity deteriorates due to inclusion of uncommitted or partially written files
Solution Approach 1:
The system implements feedback mechanisms where readers query marker records to determine the commit status of transactions before including files in directory listings. The start marker and committed marker provide feedback information about transaction state, allowing readers to make informed decisions about which files to include. This feedback loop ensures that directory listings reflect only committed transactions while maintaining information completeness for valid data.
3Productivity
If garbage collection is performed to remove uncommitted files, then storage efficiency is improved, but risk of removing committed files increases due to timing issues and race conditions
Solution Approach 1:
The garbage collection process performs preliminary actions by querying marker records to determine the commit status of transactions before removing files. The system checks for the presence of committed markers to verify that files are safe to delete. This preliminary verification mechanism ensures that only uncommitted files are removed, maintaining data integrity while improving storage efficiency by cleaning up intermediate results from failed or aborted transactions.
Data Source
AI summary
A system for directory level atomic commits includes an interface and a processor. The interface is configured to receive an indication to provide a set of files. The processor is configured to determine whether a file in a directory has been either 1) atomically committed or 2) written by a non-atomic process and not designated as deleted and provide the file as one file of the set of files in the event that the file in the directory has been either 1) atomically committed or 2) written by a non-atomic process and not designated as deleted.