Virtual Copy Forward Garbage Collection in Cloud Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing garbage collection mechanisms in cloud computing networks face challenges with long-term retention due to high costs and latency associated with remote read/write operations, particularly in deduplicating storage systems where identifying live data and reorganizing it efficiently is complex.
Innovation Solution
The implementation of a virtual copy forward process that avoids remote read/write operations by manipulating metadata locally, using CMETA containers to manage and update references, and performing garbage collection operations within the cloud network, thereby reducing the need for costly cloud-based data transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional copy forward process is used to read compression regions from original containers and write new compression regions into new containers, then garbage collection can be performed, but the process becomes slow and expensive due to remote read/write latency in cloud storage
Solution Approach 1:
The patent segments the container structure into two distinct parts: metadata sections that are copied forward to new containers, and compression regions that remain in place in the original containers. This segmentation allows the garbage collection process to operate on metadata locally without requiring remote reads of compression regions, thereby resolving the contradiction between maintaining GC functionality and improving GC speed.
Solution Approach 2:
The patent introduces a new intermediary structure called CMETA containers that store metadata sections separately from compression regions. These CMETA containers act as mediators between the garbage collection process and the actual data storage, enabling local metadata operations while maintaining references to remote compression regions without requiring their data transfer.
2Reliability
If traditional copy forward process is used to transfer data between containers, then live data can be preserved, but the cost of cloud-based read/write operations increases significantly
Solution Approach 1:
The patent extracts the metadata section from the traditional container structure and places it in a separate CMETA container structure. This extraction allows the metadata to be manipulated locally during garbage collection without incurring cloud read/write costs, while the compression regions containing actual data remain in place and are only accessed when necessary for data retrieval, thus preserving live data at lower cost.
Solution Approach 2:
The patent performs preliminary actions by copying forward metadata sections to new CMETA containers before any garbage collection operations are needed. This preliminary setup creates a local cache of metadata that enables subsequent garbage collection operations to proceed without expensive remote reads, thereby reducing cloud operation costs while maintaining data integrity.
3Measurement precision
If remote read operations are performed to identify live segments in compression regions, then accurate garbage collection can be performed, but the time required for garbage collection increases due to network latency
Solution Approach 1:
The patent segments the container into metadata sections and compression regions, and further segments the garbage collection process into metadata processing and data validation phases. By copying forward only metadata sections to local CMETA containers, the system enables fast local processing for live segment identification, while actual data validation against compression regions occurs only when necessary, thereby reducing garbage collection time without sacrificing accuracy.
Solution Approach 2:
The patent performs preliminary copying of metadata sections to new CMETA containers before garbage collection operations. This preliminary action creates a local working copy that can be processed quickly without network latency, enabling accurate identification of live segments through local metadata operations while minimizing the need for slow remote reads during the actual garbage collection process.
Data Source
AI summary
A garbage collection (GC) process within a deduplication backup network comprising a GC component identifying metadata stored in file system (FS) segments, storing the metadata in a metadata container locally on the server as well as on cloud storage, and reading the locally stored metadata container through the GC process to obtain metadata of the FS containers and determine live data regions of the FS containers, wherein the metadata contains fingerprints of all segments of the file system containers; and a copy forward component forwarding the live data regions to new containers written both locally on the server and on the cloud storage, writing live portions of the metadata container to a new metadata container written both locally on the server and on the cloud storage, and deleting dead compression regions from the cloud storage and deleting the original metadata container from local storage and the cloud storage.