Granular Entry Self-Healing in Distributed File Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed file systems face inefficiencies in self-healing processes, particularly when a storage server goes down, as they require scanning and comparing entire directories, which is time-consuming and resource-intensive, especially when only a few files have been modified among millions.
Innovation Solution
Implementing incremental lists or indices to track specific files or directories that have been created, modified, or deleted, allowing for granular entry self-healing by propagating changes using these lists when the server is reinstated, thereby avoiding the need to compare all files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the entire directory is scanned and compared to identify modified files, then all file changes can be detected, but the process becomes time-consuming and resource-intensive when directories contain millions of files
Solution Approach 1:
The patent segments the directory scanning process by introducing incremental lists that track only specific files or directories that have been created, modified, or deleted. Instead of scanning the entire directory, the system divides the work into manageable segments represented by these incremental lists, which contain only the relevant changes that need to be propagated to the failed server's replica.
Solution Approach 2:
The patent applies preliminary action by maintaining incremental lists that continuously track file changes as they occur. Before the self-healing process needs to occur, the system has already recorded all modifications in these incremental lists, so when a server fails and comes back online, the healing process can directly apply the pre-recorded changes without needing to scan or compare files at that moment.
2Measurement precision
If the entire directory is scanned and compared to identify modified files, then all file changes can be detected, but resource consumption increases significantly
Solution Approach 1:
The patent segments the directory scanning process by introducing incremental lists that track only specific files or directories that have been created, modified, or deleted. Instead of scanning the entire directory, the system divides the work into manageable segments represented by these incremental lists, which contain only the relevant changes that need to be propagated to the failed server's replica.
Solution Approach 2:
The patent applies partial action by performing only the necessary file comparisons and propagations indicated by the incremental lists, rather than scanning and comparing all files in the directory. This partial action approach focuses computational resources only on the files that actually changed, avoiding wasteful processing of unchanged files.
3Reliability
If conventional self-healing methods are used, then replica consistency can be restored, but the process is inefficient when only a few files have been modified among millions
Solution Approach 1:
The patent applies preliminary action by maintaining incremental lists that continuously track file changes as they occur. Before the self-healing process needs to occur, the system has already recorded all modifications in these incremental lists, so when a server fails and comes back online, the healing process can directly apply the pre-recorded changes without needing to scan or compare files at that moment.
Solution Approach 2:
The patent applies self-service by enabling the distributed file system to automatically track and propagate changes without requiring manual intervention or complex coordination protocols. The incremental lists self-update as files change, and the self-healing process automatically retrieves and applies these changes when a server becomes available, making the system self-managing and highly efficient.
Data Source
AI summary
A processing device of a client of a distributed file system receives a file operation request, requests creation of a plurality of entries, wherein each entry of the plurality of entries is created in a corresponding file index in each of a set of servers in the DFS, the plurality of entries corresponding to replicas of directories and one or more files that are a subject of the file operation, each entry of the plurality of entries having a unique identifier, and requests performance of the file operation by each server in the set of servers of the DFS. The processing device further receives zero or more confirmations from the set of servers, each received confirmation corresponding to an indication that the file operation was completed by a corresponding server in the set of servers. Responsive to each of the zero or more confirmations, the processing device sends a request to a respective server to clear a respective entry from the file index corresponding to a respective confirmation. Responsive to a reboot of a first server in the set of servers, a processing device of the first server to automatically process the file index on that server by performing a corresponding file operation for each entry in the file index, wherein performing the corresponding file operation includes requesting a copy of a file indicated by each entry from a replica in the set of servers.