Back-reference Data Structure for Deduplication Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication storage systems face performance degradation due to the high number of input/output (I/O) operations required for housekeeping jobs, such as deleting backup items, which consume significant bandwidth and reduce system efficiency.
Innovation Solution
A deduplication storage system generates a housekeeping work map that processes container indexes without loading manifests into memory, using back-reference data structures to decrement reference counts for data units, thereby reducing I/O bandwidth usage and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system loads manifests into memory to process housekeeping jobs, then the completeness of data processing is improved, but the I/O bandwidth consumption and system performance degradation worsen
Solution Approach 1:
The patent extracts the essential information needed for housekeeping operations from the manifest into a back-reference data structure stored within the container index. This allows the system to perform deletion operations using only the container index and back-reference structure, eliminating the need to load the entire manifest into memory while maintaining processing completeness.
Solution Approach 2:
The patent creates a back-reference data structure that copies only the necessary reference information (data unit ranges and associated metadata) from the manifest into the container index. This copied structure suffices for housekeeping operations, reducing I/O requirements while preserving the ability to accurately identify and delete data units.
2Measurement precision
If the system performs comprehensive housekeeping operations, then the data management accuracy is improved, but the I/O operations and bandwidth consumption increase
Solution Approach 1:
The patent extracts only the critical reference information needed for accurate data management from the manifest into the back-reference data structure. This extracted information includes data unit ranges and their associations, which are sufficient to perform accurate housekeeping operations without requiring full manifest loading, thus reducing I/O bandwidth consumption.
Solution Approach 2:
The patent performs preliminary action by pre-storing the back-reference data structure in the container index during normal operations. This pre-computed structure is ready for immediate use during housekeeping operations, eliminating the need for real-time manifest loading and reducing I/O bandwidth consumption during critical deletion tasks.
Data Source
AI summary
Example implementations relate to deduplication operations in a storage system. An example includes generating a housekeeping work map to delete a backup item stored in a deduplication storage system; selecting a first work entry of the housekeeping work map, where the first work entry identifies a first container index and a first manifest; in response to a selection of the first work entry, loading the first container index into the memory, the first container index comprising a back-reference data structure; identifying, in the back-reference data structure, a back-reference entry indexed to the first manifest; determining, using the back-reference entry indexed to the first manifest, a first set of data units included in the first manifest and that are indexed in the first container index; and decrementing, in the first container index, a set of reference counts for the determined first set of data units.