Storage Bucket Scheduling with Quota-Based Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in scheduling lifecycle management tasks across different buckets due to varying numbers of expired objects and processing times, leading to unfair task processing and increased storage costs.
Innovation Solution
A resource scheduling method that involves acquiring manifest files for each bucket, allocating quotas based on preset rules, generating data processing tasks, and executing these tasks in parallel to ensure fair and timely completion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If objects in buckets are scanned and processed sequentially based on management policy, then each bucket can be processed individually, but the overall task processing efficiency is low and storage costs increase due to unfair scheduling
Solution Approach 1:
The patent merges multiple bucket processing tasks into a unified parallel processing framework. Multiple worker processes simultaneously handle different buckets or task queues, combining individual bucket processing into a coordinated system that achieves global scheduling optimization and improves overall processing efficiency
Solution Approach 2:
The patent implements dynamic task allocation and parallel processing where the system adaptively adjusts processing resources based on bucket characteristics and expiration volumes. Worker processes dynamically pick up tasks from different queues based on current system state, enabling flexible and efficient resource utilization
2Speed
If processing tasks are allocated without quota control, then processing speed may increase, but resource allocation becomes unfair and storage costs increase
Solution Approach 1:
The patent introduces quota parameters (single-bucket quota and total quota) to control task allocation. By adjusting these parameters, the system can flexibly control the number of tasks assigned to each bucket and the total processing capacity, achieving both high-speed processing and fair resource distribution through parameter-based management
3Device complexity
If all expired objects are processed in a single task queue, then simple scheduling is maintained, but processing efficiency decreases due to varying bucket characteristics
Solution Approach 1:
The patent segments the processing system into multiple worker processes that can independently handle different buckets or task queues. Each worker process maintains simple operation, but the segmented architecture enables parallel processing and global scheduling optimization, resolving the contradiction between simplicity and efficiency
Data Source
AI summary
A resource scheduling method based on a storage system, a medium, and an electronic device are provided. The method includes: acquiring a manifest file respectively corresponding to each object bucket in a storage system; performing loop traversal allocation for each object bucket based on a preset object allocation rule to obtain a target object allocation quota for each object bucket; generating a data processing task corresponding to each object bucket based on the target object allocation quota for each object bucket and the manifest file corresponding to each object bucket, where a number of expired objects in a manifest file executed by a total data processing task of an object bucket is not greater than a target object allocation quota for the object bucket; and controlling the data processing task of each object bucket to be executed in parallel.