Dynamic Resource Borrowing in Distributed File System Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed file system clusters face inefficiencies in resource management, particularly when sudden increases in data volume lead to insufficient resources, as existing expansion methods are either untimely or wasteful, and current resource scheduling methods do not effectively reuse resources among cluster instances.
Innovation Solution
A resource scheduling method and system that allows cluster instances to borrow resources from other instances when needed, with a metadata server determining and requesting borrowable data server resources, enabling dynamic resource sharing and reuse.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the expansion method is adopted to cope with sudden increase in data volume, then the storage capacity is increased, but the timeliness cannot meet real-time processing requirements and resources are wasted
Solution Approach 1:
The patent implements dynamic resource allocation by enabling cluster instances to borrow data servers from other instances in real-time based on current load conditions. The metadata server dynamically determines available resources and allocates them to cluster instances that need them, allowing the system to adapt flexibly to sudden data volume increases without pre-expansion delays
Solution Approach 2:
The patent makes data servers universal by allowing them to serve multiple cluster instances. A data server can be dynamically assigned to different cluster instances based on demand, transforming static dedicated resources into flexible shared resources that can be reallocated across the distributed file system cluster
2Quantity of substance
If the expansion method is adopted to cope with sudden increase in data volume, then the storage capacity is increased, but a lot of resources are wasted
Solution Approach 1:
The patent implements resource recovery by allowing cluster instances to return borrowed data servers when they no longer need them. The metadata server tracks borrowed resources and manages their return to the original cluster instance or reallocation to other needy instances, ensuring resources are continuously utilized rather than wasted
Solution Approach 2:
The patent changes the state of data server allocation from static to dynamic by introducing borrowing and returning mechanisms. The availability and assignment of data servers are continuously adjusted based on real-time monitoring of cluster instance needs, optimizing resource utilization parameters
3Quantity of substance
If separate expansion is adopted to increase resources, then the storage capacity is increased, but resource utilization rates are not optimized
Solution Approach 1:
The patent merges resource pools across multiple cluster instances by allowing them to share data servers through the borrowing mechanism. This consolidation transforms isolated siloed resources into a unified pool that can be efficiently allocated based on overall system needs rather than individual instance constraints
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Provided are a resource scheduling method, apparatus, device and system. The method comprises: if the number of available resources owned by a first cluster instance is less than a threshold value, the first cluster instance sending a resource borrowing request to a second cluster instance; receiving borrowable resource information returned by the second cluster instance, wherein the borrowable resource information represents borrowable data server resources in the second cluster instance; and the first cluster instance acquiring the borrowable data server resources for data processing, thus realizing resource reuse between cluster instances, while also making full use of the resources of each cluster instance, and improving the resource utilization rates of the clusters.