Difference-Level Snapshots in Distributed Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data center systems face challenges in achieving high storage efficiency, high system bandwidth, and low-latency data recovery while maintaining fault tolerance, as optimizing one aspect often compromises performance in others.
Innovation Solution
The implementation of a method for efficient storage and recovery of difference-level snapshots in a distributed computing system, where data objects are subdivided into chunks, and metadata is updated to indicate associations between chunk keys and values, enabling fault-tolerant uploads and quick recovery by encoding mappings in compact self-balancing data trees.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored using conventional methods in distributed systems, then storage capacity is achieved, but storage efficiency and bandwidth utilization are compromised
Solution Approach 1:
The patent segments data objects into smaller data blocks and further divides them into data chunks. This segmentation enables efficient storage by allowing selective transmission and storage of only necessary data portions, improving storage efficiency while maintaining capacity. The segmented structure facilitates parallel processing and bandwidth optimization in distributed systems.
Solution Approach 2:
The patent introduces a hierarchical dimension to data storage by organizing data into multiple levels: data objects, data blocks, and data chunks. This multi-dimensional organization allows the system to optimize storage efficiency at each level independently, resolving the contradiction between storage capacity and efficiency by operating at different granularities simultaneously.
2Reliability
If conventional storage methods are used, then data can be stored, but data recovery latency is high and fault tolerance is compromised
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing metadata that maps data blocks to their locations and relationships before failures occur. The key-value store pre-organizes chunk mappings and data block associations, enabling immediate recovery operations without computation delays during fault events, thus reducing recovery latency while maintaining fault tolerance.
Solution Approach 2:
The patent creates redundant copies of data chunks across multiple storage nodes in the distributed system. These copies are managed through the key-value store which tracks multiple references to the same data blocks. When failures occur, the system can immediately retrieve data from alternative copies without lengthy recovery processes, reducing latency while ensuring fault tolerance.
3Reliability
If snapshots are stored completely without deduplication, then data integrity is maintained, but storage efficiency decreases
Solution Approach 1:
The patent extracts only the unique portions of data between snapshots by comparing data blocks across different versions. The system identifies and stores only the changed data blocks in difference-level snapshots, while maintaining references to unchanged blocks from previous snapshots. This extraction approach preserves data integrity for all snapshots while significantly improving storage efficiency by eliminating redundant data storage.
Solution Approach 2:
The patent changes the parameter of snapshot storage from complete copies to difference-level representations. By storing snapshots as differences relative to previous versions rather than complete copies, the system maintains data integrity through the ability to reconstruct any snapshot while improving storage efficiency. The key-value store manages these parameter changes by tracking which data blocks have changed between snapshot versions.
Data Source
AI summary
Techniques for the increased efficiency of storing data objects storage in the object storage of a software designed data center (SDDC) are provided. The techniques include the efficient storage of data, while enabling snapshots of each updating of the data. The snapshots of the data may be efficiently recovered via the techniques. Difference-level mappings for each snapshot are encoded in compact self-balancing data trees included in the object's metadata. The metadata mappings include mappings between various address spaces employed by the SDDC, as well as the address spaces employed by data stores that store the data on physical medium. Because the metadata is efficiently structured, the metadata for an object may be cached for quick lookups during data access and/or snapshot recovery. The techniques also provide low-latency recovery and/or system rollback in the event of any failure in the SDDC.