Parallel Filesystem Tree Traversal via Dynamic Workload Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traversing extremely large filesystems imposes a heavy processing load due to the need for directory traversal, making existing methods inefficient for large volumes of data.
Innovation Solution
Implementing a parallel Depth-first search (DFS) method that distributes the traversal of a filesystem tree among multiple compute nodes, allowing for accelerated traversal and workload balancing without prior knowledge of the directory tree structure, using a shared memory system to reassess and redistribute subdirectories among nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single processing node traverses the entire filesystem tree sequentially, then the traversal is simple to implement, but the processing time becomes excessively long for extremely large filesystems
Solution Approach 1:
The filesystem tree traversal is divided into multiple independent sub-tasks that can be processed in parallel. Each processing node handles a portion of the directory tree, allowing simultaneous traversal of different branches. This segmentation enables the system to process extremely large filesystems by distributing the traversal workload across multiple nodes, thereby increasing traversal speed without requiring complex coordination mechanisms.
2Loss of time
If multiple compute nodes are used for parallel traversal, then the traversal time is reduced, but the communication overhead and coordination complexity increase
Solution Approach 1:
Each processing node autonomously manages its own traversal tasks and dynamically requests additional subdirectories from the shared memory system when its local queue is exhausted. This self-service mechanism eliminates the need for centralized task assignment and reduces communication overhead, as nodes independently make decisions about their workload without requiring constant coordination with other nodes or a central controller.
Solution Approach 2:
A shared memory system acts as an intermediary between processing nodes, storing the filesystem tree structure and enabling nodes to independently retrieve and process subdirectories. This intermediary approach allows parallel traversal by providing a common data source that all nodes can access without direct node-to-node communication, thereby reducing communication complexity while maintaining parallel processing efficiency.
3Productivity
If the workload is statically distributed among compute nodes, then the implementation is straightforward, but load balancing is poor when the directory tree structure is unknown or unbalanced
Solution Approach 1:
The workload distribution is made dynamic through a shared memory system that allows processing nodes to continuously monitor and adjust their task queues. When a node completes its current subdirectories, it dynamically requests additional work from the shared memory, ensuring that workload is redistributed based on actual processing capacity and remaining tree structure. This dynamic approach automatically adapts to unbalanced or unknown directory structures, maintaining optimal load balancing without requiring prior knowledge of the filesystem topology.
Data Source
AI summary
A method for traversal of a filesystem tree, the method may include traversing the filesystem tree by multiple processing entities of a set of processing entities that belong to a storage system; wherein the traversing comprises multiple iterations of on-the fly allocation of workload, associated with parallel traversing of the filesystem tree, among the multiple processing entities; wherein a current iteration of the on the fly allocation is (a) executed by a current group of processing entities that are currently assigned to traverse current nodes of the filesystem tree, and (b) comprises re-allocating by the current group, a traversal task for traversing one or more child nodes of each of the current nodes of the filesystem tree, to a next group of processing entities; wherein the current group and the next group belong to the set.