Parallel Incremental Backup Using Bucket Subgroup Prefetching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Amazon S3 API for listing objects in a bucket is single-threaded, leading to bottlenecks and slow performance when performing incremental backups of large object storage buckets, especially when change tracking is not available, resulting in prolonged backup times.
Innovation Solution
The technique involves dividing the object storage bucket into subgroups using partition identifiers derived from checkpoint files, assigning prefetchers to each subgroup to traverse and identify differences in parallel, and ingesting new data in parallel, ensuring entries are sorted lexicographically to facilitate efficient incremental backups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the S3 API is used to list objects in a bucket, then the backup process can identify objects to backup, but the single-threaded API calls cause bottlenecks and slow performance
Solution Approach 1:
The patent divides the object storage bucket into multiple subgroups using partition identifiers, allowing parallel processing of object listing operations. Each thread handles a specific subgroup, transforming the single-threaded sequential API calls into multi-threaded parallel operations, thereby resolving the bottleneck and improving backup speed.
Solution Approach 2:
The patent introduces a partitioning dimension to the traditional sequential object listing process. By adding partition identifiers and organizing objects into subgroups across multiple dimensions, the system enables concurrent API calls from different threads, effectively adding a parallel processing dimension to overcome the single-threaded limitation.
2Quantity of substance
If the API is called repeatedly to obtain all objects, then complete object listing is achieved, but the cumulative time consumption increases significantly
Solution Approach 1:
The patent segments the complete object listing task into multiple smaller tasks by dividing objects into subgroups. Each thread performs fewer API calls for its assigned subgroup, and these operations execute concurrently. This segmentation strategy reduces the total time consumption while maintaining complete object processing.
Solution Approach 2:
The patent ensures continuous useful action by having multiple threads simultaneously performing object listing operations on different subgroups. While one thread is processing its subgroup, other threads are concurrently processing their respective subgroups, eliminating idle waiting time and maintaining continuous productive work throughout the backup process.
Data Source
AI summary
An assigned subgroup that includes a plurality of entries is traversed by a prefetcher. It is determined that an expected number of entries associated with the assigned subgroup have been traversed. In response to determining that expected number of entries associated with the assigned subgroup have been traversed, it is determined that a last read entry associated with the assigned subgroup does not correspond to a last entry associated with the assigned subgroup. The prefetcher is preempted by stopping the prefetcher from obtaining a list of entries associated with a remaining portion of the assigned subgroup.