Dynamic Put Operation Sizing for Cloud Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage providers charge for data access, leading to high costs and latency issues during data backup due to inefficient management of put operations and data object sizes, necessitating a balance between cost and speed.
Innovation Solution
Optimizing the number of put operations and size of data objects by determining the maximum object size, maximum number of threads for simultaneous transfer, and bandwidth, using multi-part, multi-threaded put operations to minimize costs while ensuring timely backups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple small put operations are used to write data objects to cloud storage, then data backup can be performed with smaller data objects, but the number of put operations increases leading to higher costs and increased latency
Solution Approach 1:
The system dynamically determines the optimal number of parts to divide each data object into, based on current stream size, bandwidth conditions, and cloud storage configuration. This dynamic adjustment allows the system to adapt to varying data sizes and network conditions, optimizing the balance between put operation count and backup speed without manual intervention.
Solution Approach 2:
The system changes key parameters including part size, number of parts per data object, and number of concurrent threads based on stream size and bandwidth measurements. By adjusting these parameters dynamically, the system optimizes put operation efficiency, reducing both latency and cost while maintaining adaptability to different backup scenarios.
2Productivity
If maximum number of threads are used for simultaneous data transfer, then backup speed increases, but resource consumption and system complexity increase
Solution Approach 1:
The system determines an optimal number of threads that is sufficient to achieve high backup speeds without over-provisioning resources. By calculating the appropriate thread count based on stream size and bandwidth, the system avoids the complexity of managing excessive threads while still achieving near-maximal transfer performance.
3Quantity of substance
If larger data objects are created from the data stream, then the number of put operations decreases reducing cost, but the size of each object increases potentially exceeding cloud storage limits
Solution Approach 1:
The system segments each data object into multiple parts that can be transmitted through multiple threads simultaneously. This segmentation allows the system to maintain data objects within cloud storage size limits while still achieving high throughput through parallel transmission, and the number of segments is optimized to minimize the total number of put operations.
Data Source
AI summary
A plurality of data objects is generated from a data stream received for constructing a backup image. A stream size of the data stream is determined. For each multi-threaded put operation used to write a data object of the plurality of data objects to a cloud storage, a maximum number of threads is determined into which the data object can be separated into a number of parts for simultaneous transmission to the cloud storage. A part size of each part is determined. A number of multi-threaded put operations is determined for writing the plurality of data objects to the cloud storage based, at least in part, on the stream size, the maximum number of threads into which the data object can be separated, and the part size. The plurality of data objects is stored into the cloud storage using the number of multi-threaded put operations.