Housekeeping Work Map for Deduplication Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication storage systems face inefficiencies during housekeeping operations due to high I/O bandwidth consumption and processing time when loading and saving container indexes multiple times, which affects data deduplication performance.
Innovation Solution
A housekeeping work map is used to accumulate update information for deduplication storage systems, allowing for the selection and processing of container indexes with the largest estimated number of updates, thereby reducing I/O costs and improving throughput by consolidating updates across multiple manifests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If container indexes are loaded and saved multiple times during housekeeping operations, then data integrity is maintained, but I/O bandwidth consumption increases and processing time is extended
Solution Approach 1:
The patent applies preliminary action by pre-loading container indexes into memory before housekeeping operations begin, and pre-allocating work entries in the work map data structure. This preparation eliminates the need for repeated loading and saving operations during actual housekeeping, reducing I/O bandwidth consumption and processing time while maintaining data integrity through proper reference count management.
2Reliability
If container indexes are loaded and saved multiple times during housekeeping operations, then data integrity is maintained, but I/O bandwidth consumption increases
Solution Approach 1:
The patent merges multiple container index operations into a single unified work map data structure that tracks all housekeeping operations. By consolidating update information for multiple container indexes into work entries that accumulate reference count changes, the system performs one I/O operation instead of multiple separate operations, significantly reducing I/O bandwidth consumption while maintaining data integrity.
3Ease of manufacture
If container indexes are processed individually during housekeeping, then processing is simple, but throughput is reduced
Solution Approach 1:
The patent segments housekeeping operations into discrete work entries in the work map, where each work entry represents a unit of work for a specific container index. This segmentation allows the system to process multiple container indexes in parallel by assigning different work entries to different processing threads, thereby increasing throughput while maintaining processing simplicity through the structured work entry format.
Solution Approach 2:
The patent implements dynamic processing by allowing the work map to adaptively manage work entries based on current system state. The work entries can be processed in any order and can be dynamically added or removed from the work map, enabling flexible parallel processing that maximizes throughput while keeping the processing logic simple and modular.
Data Source
AI summary
Example implementations relate to deduplication operations in a storage system. An example includes initiating a housekeeping job to delete a backup item stored in a deduplication storage system; identifying a set of container indexes affected by the housekeeping job, the set of container indexes including at least a first container index; in response to a determination that a housekeeping work map includes a first work entry associated with the first container index, determining an ingest count of new data units added to a first container during an ingest of the backup item into the deduplication storage system, the first container associated with the first container index; and adding the determined ingest count to the cumulative counter included in the first work entry associated with the first container index.