Bucket Inventory Task Splitting for Fair Object Storage Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing object storage bucket inventory services face challenges in ensuring timeliness and fairness due to varying data amounts across storage buckets, with large data buckets occupying excessive resources and delaying smaller data buckets.
Innovation Solution
The bucket inventory batch processing tasks are split into sub-tasks based on a preset data amount, prioritized by execution order, and distributed fairly across storage buckets to ensure each bucket receives adequate resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch processing tasks are processed sequentially without splitting, then task execution simplicity is maintained, but storage buckets with large data amounts occupy excessive resources and delay smaller data buckets
Solution Approach 1:
The patent divides each batch processing task into multiple sub-tasks based on data amount thresholds. Tasks are segmented into first batch processing sub-tasks (for data exceeding threshold) and second batch processing sub-tasks (for data within threshold), enabling fairer resource distribution across storage buckets of different sizes while maintaining manageable task complexity through structured segmentation rules
2Reliability
If tasks are split into multiple sub-tasks, then resource distribution fairness is improved, but task management complexity increases
Solution Approach 1:
The patent implements dynamic task splitting based on real-time data amount assessment. The system dynamically determines whether to create first or second batch processing sub-tasks based on the actual data quantity in each storage bucket, allowing flexible adaptation to varying data sizes while managing scheduling complexity through condition-based decision logic
3Productivity
If sequential batch processing is used, then implementation simplicity is maintained, but timeliness of processing for all buckets cannot be ensured
Solution Approach 1:
By segmenting large tasks into multiple sub-tasks, the system enables parallel execution of smaller units, reducing overall processing time and ensuring timeliness for all storage buckets regardless of their data volume, while maintaining implementation simplicity through standardized segmentation criteria
Data Source
AI summary
A processing method of an object storage service batch processing task, an electronic apparatus and a storage medium are provided. The processing method includes: obtaining a bucket inventory batch processing task corresponding to each bucket inventory rule by using a bucket inventory service for an object storage; performing a task splitting respectively for each bucket inventory batch processing task based on a preset data amount, to obtain a plurality of batch processing sub-tasks corresponding to each bucket inventory batch processing task and having an execution priority order; and selecting a plurality of batch processing sub-tasks in sequence based on a bucket order of a storage bucket and the execution priority order, and triggering selected batch processing sub-tasks to be allocated and executed.