Inline Housekeeping for Deduplication Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication storage systems face performance degradation due to the high I/O bandwidth required for housekeeping tasks, such as decrementing reference counts for deleted backup items, which involves loading and unloading metadata from persistent storage.
Innovation Solution
Implementing inline housekeeping during data ingest operations, where the controller loads a container index into memory and performs housekeeping tasks, such as decrementing reference counts, while the index is already loaded, eliminating the need for separate I/O operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate housekeeping tasks are performed after data ingest, then metadata consistency is maintained, but I/O bandwidth consumption increases and system performance degrades
Solution Approach 1:
The patent combines housekeeping tasks with data ingest operations by performing both simultaneously during the same I/O operation. The controller loads container indexes into memory during data ingest and executes housekeeping operations (decrementing reference counts, removing deleted backup items) on the loaded indexes without requiring separate I/O operations, thus merging two previously separate processes into one
Solution Approach 2:
The patent performs housekeeping operations preliminarily during data ingest before the system would normally perform them separately. By loading container indexes into memory during data ingest and executing housekeeping tasks on these loaded indexes, the system prepares and maintains metadata consistency in advance, eliminating the need for subsequent separate housekeeping I/O operations
2Speed
If container indexes are loaded into memory for data ingest operations, then data processing speed improves, but memory usage increases
Solution Approach 1:
The patent makes the loaded container indexes serve multiple functions simultaneously. The same container indexes loaded into memory for data ingest operations are also used to perform housekeeping tasks (decrementing reference counts, removing deleted items). This multi-functional use of the loaded indexes maximizes the utility of the memory resources consumed during data ingest
Data Source
AI summary
Example implementations relate to deduplication operations in a storage system. An example includes receiving a stream of data units to be stored in a persistent storage of a deduplication storage system; loading a first container index into the memory to index one or more data units included in the stream; updating the first container index to indicate a storage location of a first data unit included in the stream; reading a manifest list in the first container index to identify a first manifest indexed by the first container index; and, while the first container index remains loaded in the memory to index the one or more data units, in response to a determination that the identified first manifest is included in the first backup item that has been marked for deletion, decrementing a reference count of a second data unit.