Object Storage Batch Task Splitting for Fair Bucket Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing object storage service batch processing tasks face challenges in ensuring timeliness and fairness due to varying data amounts across storage buckets, with large data buckets occupying excessive resources and delaying smaller data buckets.
Innovation Solution
The method involves splitting bucket inventory batch processing tasks based on preset data amounts and execution priority orders, ensuring fair distribution and allocation of tasks across storage buckets with different data levels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch processing tasks are executed sequentially without splitting, then task execution simplicity is maintained, but resource utilization efficiency deteriorates and processing time increases
Solution Approach 1:
The patent divides large batch processing tasks into multiple sub-tasks based on data amount thresholds. Each sub-task is independently manageable and can be processed in parallel or sequentially without affecting the entire task workflow. This segmentation enables better resource utilization while maintaining manageable task complexity through structured sub-task organization.
2Reliability
If large data buckets are processed first, then processing completeness is ensured, but timeliness for small data buckets deteriorates
Solution Approach 1:
The patent applies different processing strategies to different data buckets based on their characteristics. Small data buckets receive prioritized processing with guaranteed resource allocation, while large data buckets are divided into sub-tasks that can be processed in parallel. This local quality approach ensures timeliness for small buckets while maintaining overall processing completeness.
3Productivity
If unlimited resources are allocated to large data buckets, then processing capability is maximized, but fairness to small data buckets deteriorates
Solution Approach 1:
The patent implements dynamic resource allocation where resource distribution adjusts based on data bucket characteristics and current system state. Small data buckets receive guaranteed minimum resources for timely processing, while large data buckets utilize remaining resources through parallel sub-task execution. This dynamic approach maximizes overall processing capability while ensuring fairness through adaptive resource management.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
A processing method of an object storage service batch processing task, an electronic apparatus and a storage medium are provided. The processing method includes: obtaining a bucket inventory batch processing task corresponding to each bucket inventory rule by using a bucket inventory service for an object storage; performing a task splitting respectively for each bucket inventory batch processing task based on a preset data amount, to obtain a plurality of batch processing sub-tasks corresponding to each bucket inventory batch processing task and having an execution priority order; and selecting a plurality of batch processing sub-tasks in sequence based on a bucket order of a storage bucket and the execution priority order, and triggering selected batch processing sub-tasks to be allocated and executed.