Cloud Backup Metadata Recovery via Object Storage Logs

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Cloud block storage platforms offer lower durability compared to cloud object storage, making metadata stored in these platforms more vulnerable to data loss, which can render snapshot backups unusable if metadata databases are corrupted or lost.

Innovation Solution

A cloud object storage-based data backup system that uploads snapshots as log segments with embedded metadata, uses a hybrid asynchronous/synchronous approach to replicate transaction logs, and includes metadata recovery processes to rebuild metadata databases from log segments and replicated transaction logs in case of failures.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If metadata is stored in cloud block storage platform, then snapshot management operations can be executed more efficiently, but durability of metadata decreases making it more vulnerable to data loss

Engineering Contradiction:

Improvesnapshot management operations efficiencyVSAvoidmetadata durability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent applies preliminary action by replicating the metadata database to a secondary location before failures occur. The system continuously synchronizes metadata between primary and secondary storage systems, ensuring that if the primary metadata database is lost, the secondary copy can be restored immediately, thus preventing data loss while maintaining efficient metadata management

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements beforehand cushioning by establishing a redundant metadata storage system with automatic failover capabilities. The system prepares backup metadata structures and replication mechanisms in advance, creating a protective buffer against durability issues in the primary cloud block storage platform, thereby maintaining reliability without sacrificing operational efficiency

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

2Device complexity

If metadata databases are stored in cloud block storage, then system complexity is reduced, but loss of metadata renders snapshot backups unusable

Engineering Contradiction:

Improvestorage system structureVSAvoidmetadata loss

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The patent applies copying by creating redundant copies of the metadata database in multiple storage locations. The system maintains primary and secondary metadata databases that are continuously synchronized, ensuring that if one copy is lost, other copies preserve the necessary metadata information, thus preventing complete data loss while managing system complexity through standardized replication protocols

Inventive Principle:

Principle #26Copying

Data Source

PatentUS11663160B2Recovering the metadata of data backed up in cloud object storage

Publication Date: 2023.05.30 VMWARE INC

AI summary

In one set of embodiments, a computer system can create a snapshot of a data set, where the snapshot includes a plurality of data blocks of the data set. The computer system can upload the snapshot to a cloud object storage platform of a cloud infrastructure, where the snapshot is uploaded as a plurality of log segments conforming to an object format of the cloud object storage platform, and where each log segment includes one or more data blocks in the plurality of data blocks, and a set of metadata comprising, for each of the one or more data blocks, an identifier of the data set, an identifier of the snapshot, and a logical block address (LBA) of the data block. The computer system can then communicate the set of metadata to a server component running in a cloud compute and block storage platform of the cloud infrastructure.