Distributed Batch Processing for Non-Uniform Object Restore
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based object storage systems face challenges in achieving uniform work distribution during object restore operations due to non-accessible partition information, leading to inefficient processing and longer times for completing tasks.
Innovation Solution
A two-stage processing method is introduced, where a distribution component receives a list of buckets, determines the number and size of restorable objects in each bucket, generates batches of uniform size, and distributes these batches among nodes for storage-related task processing, optimizing workload distribution and processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If simple partition of distributed buckets is used for object restore, then the framework can operate without accessing remote partition information, but the work distribution among nodes becomes non-uniform leading to inefficient processing
Solution Approach 1:
The system performs preliminary actions by first distributing buckets to nodes for exploration, collecting metadata about object counts and sizes before the actual restore operation. This preliminary metadata collection enables subsequent uniform partitioning of work, resolving the contradiction by preparing necessary information in advance that would otherwise require complex remote access during the restore operation itself.
Solution Approach 2:
The restore operation is segmented into distinct phases: metadata collection phase where buckets are explored and object information is gathered, and execution phase where uniform partitions are created based on collected metadata. This segmentation allows the system to maintain operational simplicity while achieving processing efficiency through informed work distribution.
2Productivity
If uniform work distribution is achieved through metadata collection, then processing efficiency improves, but the system complexity increases due to additional distribution component functionality
Solution Approach 1:
The distribution component is designed with multi-functionality, serving both as a simple bucket distributor and as a metadata collection coordinator. By integrating these functions into a single component rather than adding separate systems, the patent achieves uniform work distribution for improved processing efficiency while minimizing the increase in overall system complexity.
Solution Approach 2:
The distribution component acts as an intermediary between the restore framework and remote storage buckets. It mediates the complexity by handling all metadata collection and partitioning logic centrally, allowing the rest of the system to remain simple while still achieving efficient uniform work distribution across nodes.
3Manufacturing precision
If all bucket metadata is collected before distribution, then uniform partitions can be created, but the loss of time increases due to the two-stage processing requirement
Solution Approach 1:
The processing is segmented into two independent stages: metadata collection stage and restore execution stage. This segmentation allows parallel processing where multiple buckets can be explored simultaneously to collect metadata, and once collected, partitions can be uniformly created and distributed. The time loss is minimized by performing metadata collection operations in parallel rather than sequentially.
Solution Approach 2:
The system maintains continuity of useful action by overlapping metadata collection with prepare work. While metadata is being collected from remote buckets, the distribution component can simultaneously plan partition structures and prepare distribution strategies, ensuring that no time is wasted idle and the two-stage process flows continuously rather than with gaps.
Data Source
AI summary
The described methods, systems, and other aspects can advantageously provide balanced multi-stage processing of non-uniform object data. An example method may receive a list of buckets. Each of the buckets in the list of buckets can store one or more restorable objects. The method further comprises distributing the list of buckets to the two or more second nodes; determining a number of the one or more restorable objects in each bucket; determining a size of the one or more restorable objects in each bucket; generating batches of to-be-restored data objects based on the determined number of the one or more restorable objects in each bucket and the determined size of the one or more restorable objects in each bucket; and distributing the batches among the two or more second nodes for storage-related task processing.