Tiered Storage for Archived Object Snapshots
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large data archives for backup and disaster recovery in software-defined data centers is inefficient due to varying storage costs, access speeds, and storage constraints from versioned objects like virtual machine disks, which require effective schema management to optimize access time, resource utilization, and financial costs.
Innovation Solution
A tiered storage structure is implemented, comprising a log-structured file system for data and content-addressable storage with a logical layer, using a segment usage table for segment cleaning, which improves efficiency and cost-effectiveness by deduplicating data and optimizing storage usage through a log-structured merge-tree.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional storage solutions are used for archiving large amounts of data, then storage capacity is provided, but access time increases and resource utilization decreases
Solution Approach 1:
The storage system is segmented into multiple tiers with different performance characteristics. Hot data that requires fast access is stored in high-performance storage tiers, while cold data is stored in lower-performance tiers. This segmentation allows the system to provide large total storage capacity while maintaining fast access times for frequently accessed data.
Solution Approach 2:
Different portions of the storage system are assigned different qualities or performance levels based on local requirements. Each tier is optimized for its specific use case, with higher tiers providing faster access for critical data and lower tiers providing economical storage for less frequently accessed data, thereby resolving the contradiction between speed and storage capacity.
2Speed
If high-performance storage solutions are used, then access speed improves, but financial costs increase
Solution Approach 1:
The storage system dynamically adjusts data placement across tiers based on access patterns and performance requirements. When data is accessed frequently, it is automatically moved to higher-performance tiers. When access frequency decreases, data is moved to lower-cost tiers. This dynamic behavior allows the system to maintain required access speeds while minimizing financial costs by avoiding over-provisioning of expensive storage resources.
Solution Approach 2:
The system changes the performance parameters of storage resources by adjusting which data resides in which tier based on workload characteristics. This allows the same physical storage infrastructure to deliver varying levels of performance at different cost points, resolving the contradiction between access speed and financial cost.
3Reliability
If versioned objects like virtual machine disks are stored, then data integrity is maintained, but storage constraints and management complexity increase
Solution Approach 1:
The tiered storage system provides a universal platform that handles multiple types of versioned objects (virtual machine disks, databases, files) through a common architecture. The system automatically manages versioning, deduplication, and tiering for all object types, reducing management complexity while maintaining data integrity across diverse workloads.
Solution Approach 2:
The storage system performs self-service operations including automatic data placement, tier migration, deduplication, and version management. These automated functions reduce the burden on administrators while maintaining data integrity, resolving the contradiction between reliability and management complexity.
Data Source
AI summary
Solutions for managing archived storage include receiving, at a first node, a snapshot comprising object data (e.g., a virtual machine disk snapshot) from a second node (e.g., a software defined data center), and storing the snapshot in a tiered structure that includes a data tier and a metadata tier. Snapshots may be used for fail-over operations and/or backups, to support disaster recovery. The data tier comprises a log-structured file system (LFS), and the metadata tier comprises a content addressable storage (CAS) identifying addresses within the LFS. The metadata tier also comprises a logical layer indicating content in the CAS. Segment cleaning of the data tier is performed using a segment usage table (SUT). Some examples include performing a fail-over operation from the second node to a third node using at least the stored snapshot for workload recovery. In some examples, the CAS comprises a log-structured merge-tree (LSM-tree).