Dynamic Thread Pool for Filesystem Backup Slicing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup systems are not optimally suited for resource utilization, often leading to suboptimal performance due to static resource consumption and potential overloading or underloading of systems, especially when dealing with large filesystems.
Innovation Solution
Implementing a dynamic thread pool based on available hardware resources to efficiently crawl and slice data, allowing for the creation of uniformly sized slices that can be backed up using multiple threads, thereby optimizing resource utilization and avoiding system overload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a static thread pool with fixed number of threads is used, then the backup process can be simplified to manage, but resource utilization becomes suboptimal and system overload may occur
Solution Approach 1:
The patent implements a dynamic thread pool where the number of threads is not fixed but can be adjusted based on system conditions. The thread pool size is dynamically modified during the backup process to match actual system resource availability and workload requirements, resolving the contradiction between management simplicity and resource utilization efficiency.
Solution Approach 2:
The patent changes the parameter of thread pool size dynamically during execution. Instead of using a fixed number of threads, the system monitors available system resources and adjusts the thread pool size accordingly, allowing optimal resource utilization while maintaining manageable complexity through automated adjustment mechanisms.
2Device complexity
If a static slicing mechanism with fixed depth levels is used, then the implementation is simpler, but performance becomes non-optimal due to overloaded or underloaded systems
Solution Approach 1:
The patent implements dynamic slicing where the number of slices and their distribution across depth levels is not fixed but adapts to system conditions. The slicing mechanism dynamically adjusts based on available threads and system resources, allowing optimal performance while maintaining implementation simplicity through automated adaptation.
Solution Approach 2:
The patent incorporates feedback mechanisms where the slicing mechanism monitors system resource availability and adjusts slice creation accordingly. This feedback loop ensures that the slicing strategy adapts to actual system conditions, optimizing backup performance while keeping the mechanism relatively simple through automated feedback-driven adjustment.
3Productivity
If the thread pool size is increased to handle large filesystems, then backup speed improves, but system resources may be overloaded
Solution Approach 1:
The patent implements feedback mechanisms that monitor system resource consumption in real-time and adjust thread pool size accordingly. This allows the system to increase thread count for faster backup when resources are available while automatically reducing it to prevent overload, maintaining both speed and stability through continuous feedback-driven adjustment.
Solution Approach 2:
The patent dynamically changes the thread pool size parameter based on system conditions rather than using a fixed high number. This flexible parameter adjustment allows the system to optimize backup speed when resources permit while preventing overload by reducing thread count when system capacity is approached, resolving the contradiction between speed and stability.
Data Source
AI summary
One example method includes gathering information regarding filesystem resources, based on the system resource information, identifying a thread pool size, starting a thread pool having the thread pool size, where the thread pool size is dynamically adjustable based on changes in the system resources, crawling a filesystem and identifying crawl jobs to be performed, adding the crawl jobs to the thread pool, and performing the crawl jobs. The method may further include slicing data in one or more directories that have been crawled.