Partial In-Line Deduplication With Post-Processing For Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in data deduplication during backups, with in-line deduplication slowing data ingestion due to metadata operations and post-processing deduplication being resource-intensive and requiring re-chunking and re-writing of data.
Innovation Solution
Implementing a technique that combines partial in-line deduplication and partial post-processing deduplication, where data chunks are deduplicated during ingestion with respect to previously stored chunks, and duplicate chunks are identified and removed during post-processing, using a tree data structure to manage metadata and reduce resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If in-line deduplication is performed during data ingestion, then storage utilization is improved by removing duplicates, but data ingestion speed deteriorates due to metadata operations
Solution Approach 1:
The deduplication process is segmented into two distinct phases: in-line deduplication during data ingestion and post-processing deduplication after ingestion. This segmentation allows the system to perform lightweight deduplication operations during ingestion while deferring more intensive metadata operations to a separate post-processing phase, thereby resolving the contradiction between storage utilization and ingestion speed.
Solution Approach 2:
The system performs preliminary deduplication actions during data ingestion by comparing incoming data chunks against previously ingested chunks and maintaining a chunk metadata data structure. This preliminary action reduces the amount of duplicate data stored while minimizing the impact on ingestion speed by performing only essential deduplication operations during the ingestion phase.
2Quantity of substance
If post-processing deduplication is performed on all data chunks, then storage utilization is improved by removing all duplicates, but resource consumption increases due to re-chunking and re-writing
Solution Approach 1:
The system performs partial post-processing deduplication by selectively processing only certain data chunks rather than all chunks. This partial action approach maintains storage utilization benefits while significantly reducing resource consumption by avoiding unnecessary re-chunking and re-writing operations on already-processed or non-duplicate data.
Solution Approach 2:
The patent extracts and removes only the duplicate data chunks from storage after ingestion, rather than re-processing all data. By using the chunk metadata data structure to identify and extract only the necessary duplicate chunks for removal, the system improves storage utilization while minimizing resource consumption associated with full re-chunking and re-writing operations.
3Speed
If partial in-line deduplication is performed with respect to previously stored chunks only, then data ingestion speed is improved by reducing metadata operations, but duplicate chunks within the same backup may remain
Solution Approach 1:
The system merges in-line deduplication and post-processing deduplication into a hybrid approach. The in-line phase performs deduplication against previously stored chunks to maintain ingestion speed, while the post-processing phase performs additional deduplication to eliminate duplicates within the same backup. This combination resolves the contradiction by distributing deduplication tasks across two phases, each optimized for its specific goal.
4Productivity
If a tree data structure is used to manage metadata, then deduplication efficiency is improved by organizing chunk information, but system complexity increases
Solution Approach 1:
The system introduces a tree data structure dimension to organize chunk metadata, transforming flat metadata storage into a hierarchical structure. This dimensional change enables more efficient deduplication operations by allowing organized storage and retrieval of chunk information, with the tree structure providing natural grouping and indexing capabilities that improve deduplication efficiency despite the increased structural complexity.
Data Source
AI summary
Data is ingested from a source system. Ingesting the data includes determining corresponding chunk identifiers for a plurality of data chunks corresponding to the ingested data and for each of the plurality of data chunks, verifying whether the corresponding chunk identifier is included in a data structure tracking identifiers of data chunks that were already stored in a storage of a storage system before the data ingestion started and storing the data chunk in a storage based on the verification. After the ingesting is completed, deduplication of the ingested data chunks stored in the storage having a same chunk identifier is performed and the data structure is updated based on the deduplication.