Archive Management Agent for Cloud Object Storage Fragmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud/object storage services face challenges such as weak consistency guarantees, limited network bandwidth and high latency, lack of object locking, and consistency issues with features like glacier tiering, making it difficult to implement efficient data backup and restore systems.
Innovation Solution
The implementation of an archive management agent that uses an efficient object format and upload process, handles deletion and restoration without explicit synchronization, optimizes fragmentation, and clones objects in the standard tier to ensure data consistency when enabling glacier tiering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is archived in cloud/object storage with eventual consistency model, then storage scalability and cost efficiency are improved, but data consistency and backup reliability deteriorate
Solution Approach 1:
The system segments data into multiple objects with versioning, where each object represents a consistent snapshot. This allows the backup system to work with discrete, consistent units rather than relying on the storage system's eventual consistency model, thereby maintaining data consistency while preserving storage scalability.
Solution Approach 2:
The patent introduces an intermediary backup management system that sits between the client and cloud storage, implementing its own consistency model. This intermediary layer handles version control, object locking, and consistency guarantees independently from the underlying eventual-consistency storage system, resolving the contradiction between storage scalability and data consistency.
2Adaptability or versatility
If network bandwidth is limited for cloud access, then storage cost per unit capacity is reduced, but write speed and backup productivity deteriorate
Solution Approach 1:
The system performs preliminary actions by pre-processing data into optimized objects with metadata and version information before upload. This preparation work reduces the need for subsequent network trips for consistency checks and updates, thereby improving write speed while maintaining cost efficiency through the use of cloud storage.
Solution Approach 2:
The patent implements continuous backup operations where data is streamed and processed continuously rather than in discrete, interruptible batches. This continuous action model maximizes network utilization within bandwidth constraints and maintains steady write throughput, improving productivity without requiring excessive bandwidth.
3Quantity of substance
If glacier tiering is enabled for cost optimization, then storage cost is reduced, but data access time and restore speed deteriorate
Solution Approach 1:
The system segments data into hot, warm, and cold tiers with different storage policies. Frequently accessed or recently backed up data remains in faster storage tiers, while older archival data moves to glacier tiering. This segmentation allows cost optimization for archival data while maintaining fast restore speeds for active data, resolving the contradiction between storage cost and restore time.
Solution Approach 2:
The patent implements preliminary actions by pre-warming or pre-loading data from glacier tiering before actual restore operations. Metadata and indexing information are maintained in faster storage tiers, allowing the system to quickly locate and initiate restore operations even when actual data resides in glacier storage, thereby reducing perceived restore time.
4Reliability
If object locking mechanism is added to synchronize concurrent access, then data consistency is improved, but system complexity and device complexity increase
Solution Approach 1:
The system implements self-service object locking where each data object carries its own version metadata and locking information embedded in its structure. Objects are designed to be self-describing with version identifiers that automatically enable consistency checks without requiring external locking mechanisms, thereby maintaining data consistency while minimizing system complexity.
Data Source
AI summary
Techniques for handling fragmentation of archived data in cloud/object storage are provided. In one set of embodiments, a computer system can upload a new snapshot of a data set to the cloud/object storage, where the new snapshot comprises a plurality of data blocks, and where the new snapshot is uploaded as one or more data objects and one or more metadata objects. For each data block in the plurality of data blocks, the computer system can identify an existing data object in the cloud/object storage where the data block is currently stored. The computer system can further select, from among the identified existing data objects, a subset of the existing data objects that are part of a snapshot created in the cloud/object storage at least a predefined number of snapshots before the new snapshot, and select one or more data blocks of one or more data objects in the subset that have not be overwritten by another snapshot. The computer system can then upload the one or more data blocks as part of the new snapshot.