Storage Bucket Task Scheduling for Fair Parallel Lifecycle Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in scheduling lifecycle management tasks for buckets due to varying numbers of expired objects and processing times, leading to unfair task scheduling and increased storage costs.
Innovation Solution
A resource scheduling method and apparatus that involves acquiring manifest files for each bucket, applying preset allocation rules to determine target quotas, and executing data processing tasks in parallel to ensure fair allocation and reduce storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If objects in buckets are scanned and processed based on management policy with sequential task queue execution, then task processing can be completed, but processing efficiency is low and storage costs increase due to unfair scheduling
Solution Approach 1:
The patent segments the processing task queue into multiple sub-queues based on bucket types or priority levels. Each sub-queue is managed independently with dedicated processing resources, allowing simultaneous processing of multiple task batches. This segmentation enables parallel execution of lifecycle management tasks across different buckets, significantly improving overall processing efficiency and reducing total processing time.
Solution Approach 2:
The patent implements dynamic task allocation where processing resources are automatically adjusted based on real-time queue status, bucket priority levels, and system load conditions. The scheduling mechanism dynamically reassigns processing capacity to high-priority buckets when resources become available, ensuring that critical lifecycle management tasks are completed promptly while maintaining fair resource distribution across all buckets.
2Ease of operation
If processing tasks are executed sequentially in a single queue, then resource allocation is simple, but task scheduling is unfair and storage costs increase
Solution Approach 1:
The patent divides the single processing queue into multiple specialized sub-queues, each handling specific bucket types or priority levels. This segmentation maintains operational simplicity through standardized queue management interfaces while internally enabling fair scheduling by dedicating processing resources to different bucket categories, thereby ensuring equitable treatment of all buckets without complex manual allocation.
Solution Approach 2:
The patent introduces a scheduling intermediary layer between the task submission interface and the execution engine. This intermediary automatically distributes tasks from the input queue to appropriate sub-queues based on predefined policies and real-time conditions, maintaining ease of operation for users while ensuring fair and reliable scheduling through automated decision-making logic that considers bucket priorities and resource availability.
3Productivity
If more processing resources are allocated to buckets with more expired objects, then processing speed increases, but resource allocation becomes unbalanced and small buckets are neglected
Solution Approach 1:
The patent changes the resource allocation parameter from a purely volume-based model to a multi-dimensional model that considers bucket size, priority level, and historical processing patterns. This parameter transformation enables the system to allocate resources proportionally to ensure that small buckets receive adequate attention while large buckets receive sufficient capacity to maintain high processing speeds, achieving both productivity and allocation balance simultaneously.
Solution Approach 2:
The patent implements an equipotential resource allocation mechanism where all buckets are given equal access to processing resources through a fair scheduling policy. The system adjusts resource distribution to maintain equal processing opportunities across buckets of different sizes, preventing resource starvation of small buckets while ensuring large buckets can process their volume efficiently, thereby achieving balanced and adaptable resource allocation.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A resource scheduling method and apparatus based on a storage system, a medium, and an electronic device are provided, so that data processing tasks of different object buckets can be scheduled more fairly, and the timeliness and efficiency of task completion are improved. The method includes: acquiring a manifest file respectively corresponding to each object bucket in a storage system; performing loop traversal allocation for each object bucket based on a preset object allocation rule to obtain a target object allocation quota for each object bucket; generating a data processing task corresponding to each object bucket based on the target object allocation quota for each object bucket and the manifest file corresponding to each object bucket, where the data processing task is a task executed according to a manifest file granularity, and a number of expired objects in a manifest file executed by a total data processing task of an object bucket is not greater than a target object allocation quota for the object bucket; and controlling the data processing task of each object bucket to be executed in parallel.