Local Metadata Staging for Cloud Data Archiving
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud/object storage services face challenges in providing strong consistency for data backup and restore systems due to eventual consistency models, limited network bandwidth, and high latency, making it difficult to manage overwrites and ensure data integrity during archival processes.
Innovation Solution
Implementing a local metadata staging mechanism where snapshot data is uploaded in fixed-sized chunks, with metadata staged locally in a B+ tree structure on-premises, avoiding overwrites and optimizing bandwidth usage by batching data and metadata uploads, and using one-to-one or arbitrary mapping approaches for block address allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If cloud/object storage services use eventual consistency models, then storage scalability and cost-effectiveness are improved, but data consistency and reliability for backup systems deteriorate
Solution Approach 1:
The patent introduces a local metadata staging layer as an intermediary between the client and cloud storage. This staging area maintains a local copy of metadata (including data block addresses and consistency information) that acts as a mediator to ensure read-after-write consistency without requiring the cloud storage to provide strong consistency guarantees. The local staging area resolves the contradiction by providing consistency locally while leveraging the scalable cloud storage for actual data retention.
2Speed
If network bandwidth between on-premises site and cloud storage is increased, then data archival speed is improved, but network costs and infrastructure complexity worsen
Solution Approach 1:
The patent segments the data archival process into two distinct phases: (1) data transfer phase where data blocks are uploaded to cloud storage, and (2) metadata staging phase where metadata is prepared and staged locally. This segmentation allows the system to optimize each phase independently - data transfer uses available network bandwidth while metadata operations occur locally without network dependency, thereby improving overall archival speed without proportionally increasing network infrastructure requirements.
Solution Approach 2:
The system performs preliminary actions by staging metadata locally before actual data archival operations. The local metadata staging area pre-allocates and prepares metadata structures (including data block address mappings and consistency information) before data blocks are uploaded to cloud storage. This preliminary local preparation reduces the need for repeated network operations during the archival process, improving speed without linearly increasing network infrastructure complexity.
3Loss of time
If latency from on-premises equipment to cloud storage is reduced, then write operation performance is improved, but network infrastructure costs worsen
Solution Approach 1:
The local metadata staging area serves as an intermediary that handles metadata operations locally, eliminating network latency for metadata read/write operations. While data blocks still require network transmission to cloud storage, the critical metadata operations (which determine data block addresses and track consistency) occur locally without network dependency, effectively reducing perceived write operation time without requiring low-latency network infrastructure.
4Reliability
If local metadata staging is implemented, then data consistency and archival reliability are improved, but local storage requirements and system complexity worsen
Solution Approach 1:
The patent extracts the consistency-critical metadata operations from the cloud storage system and places them in a separate local metadata staging area. This extraction isolates the consistency requirements to a local component that can provide strong consistency guarantees, while the cloud storage system continues to provide scalable object storage without strong consistency. The separation reduces system complexity by allowing each component to be optimized independently for its specific function.
Solution Approach 2:
The system applies local quality by providing different consistency characteristics to different parts of the system: the local metadata staging area provides strong read-after-write consistency for metadata operations, while the cloud object storage provides eventual consistency for data blocks. This differentiated approach allows the system to achieve archival reliability where needed (in metadata) without the overhead of strong consistency everywhere, managing complexity through targeted consistency enforcement.
Data Source
AI summary
Techniques for archiving data in cloud/object storage using local metadata staging are provided. In one set of embodiments, a computer system residing at an on-premises site comprising on-premises storage can receive a snapshot of a dataset to be archived. The computer system can package data in the snapshot into one or more fixed-size data chunks and upload the one or more fixed-size data chunks to cloud/object storage. Further, concurrently with the packaging and the uploading, the computer system can stage metadata for the snapshot in the on-premises storage. Then, upon uploading all of the data of the snapshot, the computer system can upload the metadata staged in the on-premises storage to the cloud/object storage.