Directory Attribute Updates in Distributed File Systems via Delta Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Concurrent operations on subdirectories in distributed file systems lead to transaction conflicts and low throughput due to frequent reading and writing of attribute files, exacerbated by queuing and serialization processing.
Innovation Solution
Implementing an incremental (Delta) updating method to store update information in a delta record, using a preset incremental table, and asynchronously merging this information into the attribute file to avoid conflicts, with mechanisms to enable delta recording on demand based on contention levels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If queuing and serialization processing is performed on operations of subdirectories under the same directory, then transaction conflicts are avoided, but throughput is reduced
Solution Approach 1:
The patent segments the attribute file updates by introducing a separate incremental table to store update information. Instead of serializing all operations, the system divides update operations into discrete records that can be processed independently and merged asynchronously, resolving the conflict between avoiding transactions conflicts and maintaining throughput.
Solution Approach 2:
The patent applies preliminary action by pre-storing update information in an incremental table before actually updating the attribute file. This allows operations to be queued and processed in advance, with the actual attribute file update deferred to a later merge operation, thereby avoiding immediate conflicts while maintaining processing throughput.
2Measurement precision
If frequent reading and writing of attribute file is performed, then attribute updates are reflected immediately, but transaction conflicts increase and throughput decreases
Solution Approach 1:
The incremental table serves as an intermediary between the operations and the attribute file. Instead of directly reading and writing the attribute file for every operation, the system writes to the incremental table first, then periodically merges this information back to the attribute file. This mediator layer reduces transaction conflicts while maintaining update responsiveness.
Solution Approach 2:
The patent implements periodic action by asynchronously merging incremental table records to the attribute file at scheduled intervals or when thresholds are met, rather than immediately reflecting every change. This periodic merging reduces the frequency of attribute file transactions while still ensuring updates are reflected in a timely manner.
Data Source
AI summary
The present disclosure provides an attribute updating method and apparatus for a directory in a distributed file system, a device and a storage medium, which relates to the field of computer technology and, in particular, to the field of big data and distributed file systems. A specific implementation solution is as follows: firstly, it can be determined whether the distributed system needs to update the attribute file of a certain directory through an incremental table. If it is required to update the attribute file of this directory through the incremental table, the update information required for updating the attribute file of the directory using the incremental table can be further determined. Further, the update information can be written into the increment table.