Distributed File System Consistency Check via Node Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed file systems, a single node performing consistency checks is inefficient due to the large size of expected physical metadata that exceeds memory capacity and wasteful resource utilization.
Innovation Solution
Each node in a distributed file system constructs an expected copy of physical metadata, transmits it to the corresponding nodes owning actual metadata, and compares the results, with a controller node aggregating the findings to generate an error report and monitor progress, distributing the task across multiple nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single node performs consistency checks on a distributed file system, then the check can be completed, but the memory requirements exceed the capacity of a single node and resource utilization is inefficient
Solution Approach 1:
The patent divides the consistency check task into segments distributed across multiple nodes. Each node performs checks on specific portions of the file system metadata rather than a single node handling the entire check, thereby reducing individual memory requirements while maintaining overall check completeness.
Solution Approach 2:
The patent transitions from a single-node vertical processing model to a multi-node horizontal distributed model. By adding the dimension of spatial distribution across multiple nodes, the system can handle larger metadata sets that would exceed the memory capacity of any single node.
2Reliability
If a single node performs consistency checks on a distributed file system, then the check can be completed, but resource utilization becomes inefficient
Solution Approach 1:
The consistency check workload is segmented and assigned to multiple nodes based on their local metadata ownership. This segmentation enables parallel processing across the distributed system, improving overall productivity and resource utilization efficiency while maintaining check completeness.
Solution Approach 2:
Each node performs consistency checks on metadata that it already possesses or owns locally, eliminating the need to transfer large amounts of data across the network. This self-service approach reduces network overhead and improves resource utilization by leveraging locally available resources.
3Reliability
If expected physical metadata is constructed for the entire distributed file system, then consistency can be verified, but the metadata size exceeds single-node memory capacity
Solution Approach 1:
The patent segments the expected physical metadata construction task across multiple nodes. Each node constructs expected metadata only for the file system portions it manages, reducing the metadata size any single node must handle while maintaining verification accuracy through coordinated distributed processing.
Data Source
AI summary
Exemplary methods, apparatuses, and systems include a controller node receiving a request to perform a consistency check of a distributed file system. The controller node transmits, to each of a plurality of nodes, a request for the node to use logical metadata of the distributed file system owned by the node to construct an expected copy of physical metadata mapped to the logical metadata, determine which of the plurality of nodes own actual portions of the physical metadata, transmit corresponding portions of the expected copy of the physical metadata to each of the nodes determined to own actual portions of the physical metadata, and compare expected copies of the physical metadata received from other nodes to the actual physical metadata owned by the node. The controller node receives a result of the comparison from each of the nodes, aggregates the received results, and generates an error report.