Asynchronous Cloud Data Deduplication and Garbage Collection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face inefficiencies in storage utilization due to the lack of effective deduplication and garbage collection processes, leading to unnecessary storage of duplicate data and inefficient resource allocation.
Innovation Solution
A data management device with a persistent storage system that performs deduplication and garbage collection concurrently, using meta-data to identify and delete duplicate segments in long-term retention storage while maintaining active storage meta-data, thereby optimizing storage space and resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deduplication and garbage collection are performed sequentially, then each operation can be completed thoroughly, but the total execution time increases and storage availability decreases
Solution Approach 1:
The patent combines deduplication and garbage collection into a single unified operation. The garbage collection process uses the deduplication index to identify segments that are both obsolete (no active references) and duplicate (identical content), allowing simultaneous execution of both functions without requiring separate processing passes, thereby reducing total execution time while maintaining data integrity.
Solution Approach 2:
The system performs garbage collection continuously during normal storage operations by leveraging the deduplication index that is continuously maintained. Instead of stopping operations to perform batch garbage collection, the system continuously identifies and removes obsolete segments as they become unreachable, ensuring storage efficiency without interrupting service continuity.
2Ease of operation
If meta-data is stored in LTR storage, then data retrieval is simplified, but storage space is wasted and garbage collection becomes less efficient
Solution Approach 1:
The system segments meta-data into two distinct components: the deduplication index (stored in active memory or fast storage) and the actual data segments (stored in LTR storage). This segmentation allows the index to be easily accessible for quick lookups while the actual data resides in LTR storage, optimizing both retrieval speed and storage efficiency by preventing duplication of meta-data in LTR storage.
Solution Approach 2:
The deduplication index acts as an intermediary between the retrieval system and LTR storage. Instead of storing all meta-data directly in LTR storage, the system maintains a separate index structure that maps segment identifiers to their locations and status. This intermediary layer enables efficient identification of garbage candidates without requiring meta-data to be physically present in LTR storage, thus saving space while maintaining ease of operation.
3Quantity of substance
If deduplication is performed against all stored segments, then deduplication ratio improves, but processing overhead and computational resources increase
Solution Approach 1:
Instead of performing exhaustive deduplication checks against all segments in LTR storage, the system performs partial deduplication by leveraging the deduplication index to identify only those segments that are candidates for removal (i.e., segments with no active references). This partial action approach achieves sufficient deduplication ratios without the prohibitive computational cost of comparing every segment against every other segment, thus reducing processing overhead while maintaining storage efficiency.
Data Source
AI summary
A data management device includes a persistent storage and a processor. The persistent storage includes meta-data of data stored in a long term retention (LTR) storage. The processor obtains a file storage request for a file and deduplicates the file against segments stored in the LTR storage while performing garbage collection on the LTR storage. Performing garbage collection includes deleting segments of the data stored in the LTR storage using the meta-data. The meta-data is not stored in the LTR storage.