Directory Tree Subdivision for Parallel Compute Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to efficiently subdivide and distribute unknown directory trees and object prefixes across multiple compute engines due to unknown data structures and sizes, leading to inefficient communication and prolonged task completion times.
Innovation Solution
An electronic system that enumerates file systems into ordered task lists, subdivides these lists into smaller subtasks, and distributes them to multiple compute engines, ensuring efficient workload distribution and dynamic adaptation to unknown directory and file structures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the work is subdivided into small quanta for distribution to separate computers, then communication overhead becomes significant relative to actual work, but if the work is kept as large units then most computers cannot finish their assigned work while one computer is still working
Solution Approach 1:
The system dynamically subdivides the directory tree into subtasks during traversal rather than using static pre-defined subdivisions. The master compute engine continuously monitors progress and adjusts task distribution in real-time, allowing adaptive workload management that responds to actual execution conditions and discovers data structure characteristics as they are encountered.
Solution Approach 2:
The directory tree is segmented into manageable subtasks that can be distributed across multiple compute engines. The master engine divides the traversal work into smaller units (subtasks) that represent portions of the directory tree, enabling parallel execution while maintaining manageable communication overhead through targeted task distribution.
2Adaptability or versatility
If the data structure is traversed from root nodes toward leaf nodes, then the contents are discovered progressively, but the structure and size of subtasks cannot be determined in advance for effective distribution
Solution Approach 1:
The master compute engine performs preliminary traversal of the directory tree to discover its structure and size before distributing work to worker engines. This preliminary action enables the system to understand the data organization and plan effective task distributions, avoiding the need to traverse the entire tree multiple times during execution.
Solution Approach 2:
The system dynamically adapts to unknown data structures by performing initial traversal to map the directory tree, then using this discovered structure to create optimized task distributions. The master engine continuously monitors worker progress and adjusts subsequent task assignments based on actual execution conditions and discovered data characteristics.
3Productivity
If separate computers do not share memory or disk, then communication requirements increase for task assignment, but the specific work assigned must be communicated to each computer
Solution Approach 1:
The master compute engine acts as an intermediary between the directory tree and worker compute engines. It receives the root directory reference, performs initial traversal to discover the data structure, and then distributes appropriately-sized subtasks to worker engines. This intermediary role simplifies communication by centralizing the complex traversal and distribution logic in one engine.
Data Source
AI summary
An electronic system for subdividing an unknown directory tree for execution of operations on directories and files therein has a plurality of computer engines. Each computer engine has a processor. A memory is coupled to the processor, the memory storing program instructions. The program instructions when executed by a processor of a first of the computer engines, causes the processor of the first computer engine to: enumerate a file system comprising directories and files into ordered lists of tasks; subdivide the lists of tasks into a number of subtasks, each of the subtasks being smaller in size than each of the tasks; and deliver the subtasks to a plurality of different computer engines.