Coalescing Buffer Optimizes Cloud Snapshot Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face challenges in achieving efficient snapshot recovery from cloud storage due to high latency and cost associated with small, serial reads, which impact recovery point objectives (RPOs) and throughput.
Innovation Solution
A two-phase approach is implemented, involving a dry-run phase to create a patch file image that identifies necessary data blocks and a data-transfer phase where smaller reads are coalesced into larger ones, allowing parallel downloads from the cloud, thereby reducing redundant data transfer and improving RPOs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If small serial reads are performed from cloud storage during snapshot recovery, then data can be retrieved with basic functionality, but throughput is limited and costs increase due to high latency and redundant transfers
Solution Approach 1:
The patent merges multiple small sequential reads into larger coalesced read operations. The coalescing buffer accumulates read requests and combines them into fewer, larger I/O operations that can be processed in parallel, thereby improving throughput and reducing recovery time while minimizing redundant data transfers from cloud storage
Solution Approach 2:
The patent performs preliminary actions by pre-fetching data blocks into a coalescing buffer before they are actually needed for recovery operations. This allows the system to prepare data in advance, organize it for efficient parallel processing, and reduce the actual recovery time when data is requested, addressing both throughput and time loss concerns
2Reliability
If snapshots are taken frequently to improve recovery point objectives, then data freshness improves, but the cost and impact on users increases significantly
Solution Approach 1:
The patent extracts and processes only the necessary data blocks for recovery operations rather than copying entire snapshot datasets. By identifying and retrieving only the specific data blocks needed for the recovery point objective, the system achieves reliable recovery without incurring the full cost and user impact of taking frequent complete snapshots
3Productivity
If parallel read operations are implemented to improve throughput, then data transfer speed increases, but system complexity and coordination overhead increase
Solution Approach 1:
The patent introduces a coalescing buffer as an intermediary component that manages parallel read operations. This buffer acts as a mediator between multiple read sources and the recovery process, coordinating parallel I/O operations, merging results, and handling dependencies without requiring complex distributed coordination logic across the entire system, thus enabling high throughput while controlling complexity
Data Source
AI summary
In some examples, a data management and storage (DMS) platform, comprises peer DMS nodes in a node cluster, a distributed data store comprising local and cloud storage, and at least one processor configured to perform operations in a method of creating a local consolidated patch file from a patch file chain stored in the cloud storage. The operations include, in a first dry-run phase, creating a logical patch file image of data blocks in one or more cloud patch files stored in the cloud storage; in a second data-transfer phase, downloading at least some of the data blocks from the cloud patch files identified by the logical patch file image, the second data-transfer phase comprising a coalescing operation to construct a set of coalesced reads of the data blocks; and creating and storing, in the local storage, the local consolidated patch file using the downloaded data blocks.