Multi-threaded Directory Tree Deletion via Dynamic Queue Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in efficiently managing large directory trees, particularly in handling deletion requests and maintaining queue thresholds to prevent overflow and ensure effective directory tree deletion operations.
Innovation Solution
A server configuration with a first processing device for storing deletion requests and a second processing device with programmable hardware chips, which isolates directory trees, reads objects, and manages work queues to ensure that objects are processed efficiently, either by pushing them into a work queue or removing them if the queue exceeds thresholds, and utilizes thread processes for directory tree deletion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the work queue size is increased to handle large directory trees, then the deletion capability is improved, but the system memory consumption increases and stability deteriorates
Solution Approach 1:
The patent divides the work queue into multiple segments or chunks, processing directory trees in smaller batches rather than loading everything into a single large queue. This segmentation allows the system to handle large directory trees by processing them in manageable portions, reducing peak memory consumption while maintaining deletion capability.
Solution Approach 2:
The work queue size is made dynamic rather than static, adjusting the queue capacity based on available system resources and the specific deletion task at hand. This dynamic adaptation allows the system to optimize between processing efficiency and memory consumption, preventing queue overflow while avoiding excessive memory allocation.
2Reliability
If the work queue threshold is set low to prevent overflow, then system stability is improved, but the deletion efficiency deteriorates due to frequent queue clearing
Solution Approach 1:
The system performs preliminary actions by isolating the directory tree in an inaccessible portion of the storage system before deletion begins. This preliminary isolation prevents client access conflicts and allows the deletion process to proceed without interruptions, improving both stability and efficiency by avoiding mid-process queue clearing.
Solution Approach 2:
The patent introduces an intermediary mechanism (the isolated storage portion and work queue management layer) between the client requests and the actual deletion operations. This intermediary buffers and manages the work queue, allowing the system to maintain stability thresholds while processing deletions efficiently without direct client interference.
3Speed
If multi-threading is implemented to speed up deletion, then processing speed is improved, but system resource management complexity increases
Solution Approach 1:
The deletion task is segmented into independent units of work that can be distributed across multiple threads. Each thread processes a portion of the directory tree independently, with the work queue managing task distribution. This segmentation enables parallel processing while keeping individual thread complexity manageable.
Solution Approach 2:
The work queue implements self-service mechanisms by automatically managing thread task assignment, queue monitoring, and threshold enforcement without requiring complex external resource management. The system self-regulates by pushing objects to the work queue when below threshold and clearing when above, reducing the burden on external resource management complexity.
Data Source
AI summary
Example implementations described herein include systems and methods for processing the deletion of a directory that can include one or more subdirectories. In a system including a server having a hardware-side and a software-side that manages one or more storage systems or storage devices, a client may request the deletion of a directory, whereupon the processing of the directory occurs in a work queue and subtree queue for processing subdirectories.