Distributed File System Data Block Consistency via Counter Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale distributed data processing systems, maintaining the consistency of redundant data blocks across multiple file servers is challenging, especially during writing operations and in the event of server failures, with existing methods like MD5 checks impacting performance.
Innovation Solution
A distributed file system method using a CHUNK counter value, where the file location register generates and manages counter values for each CHUNK, ensuring consistency by writing data to both main and standby servers and reconstructing abnormal CHUNKs based on maximal counter values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If general check method such as MD5 is applied to CHUNKs, then data consistency can be verified, but processing performance is severely affected
Solution Approach 1:
The patent changes the verification parameter from complex MD5 hash computation to simple counter value comparison. Instead of computing cryptographic hashes for each CHUNK verification, the system uses incremental counter values that can be compared directly, dramatically reducing verification overhead while maintaining consistency detection capability
Solution Approach 2:
The patent introduces lightweight counter values as disposable verification markers instead of using expensive MD5 hash computations. These counters are simple integer values that can be generated and compared with minimal computational cost, making verification operations inexpensive and scalable
2Reliability
If redundant backup of CHUNKs is implemented across multiple FASs, then data reliability is improved, but maintaining consistency of copies becomes difficult
Solution Approach 1:
The patent implements a feedback mechanism where the FLR collects counter values from multiple FASs holding CHUNK copies, compares them, and identifies inconsistencies. This feedback loop enables automatic detection and reconstruction of inconsistent copies without complex manual coordination between distributed servers
Solution Approach 2:
The patent introduces the FLR as an intermediary that coordinates consistency management across distributed FASs. Instead of requiring direct peer-to-peer coordination between multiple FASs, the FLR acts as a central mediator that collects status information, makes consistency decisions, and manages reconstruction operations
3Reliability
If data is written to both main and standby FASs simultaneously, then data safety is improved, but ensuring simultaneous writing consistency becomes complex
Solution Approach 1:
The patent applies preliminary action by having the FLR pre-allocate counter values for CHUNKs before writing operations begin. This allows the system to establish a known reference state in advance, making it easier to detect and resolve any inconsistencies that occur during parallel writing to main and standby FASs
Data Source
AI summary
A distributed file system and a data block consistency managing method thereof are disclosed. The method comprises: a file location register generates the values of the counters corresponding to CHUNKs and the values of the counters are simultaneously stored in file access servers and a file location register; when writing data into a CHUNK, a file access client writes data into both the main and standby file access servers and revises the values of counters of CHUNKs in the file access servers into which data is written normally; the file location register takes the CHUNK whose counter has the maximal value as the normal and valid one according to the corresponding values of the counters of corresponding CHUNK reported by the main and standby file access servers.