Segmented Reference Databases for Online Deduplication Repair
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional deduplication storage systems face data loss and storage leaks due to corrupted reference databases, and taking them offline to repair may not meet service level requirements.
Innovation Solution
Maintaining multiple reference databases, each storing information for different subsets of data containers, allowing for inline reference updates and online checking to repair corrupted databases while keeping the system accessible.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single reference database is used to manage deduplication references, then the system structure is simple, but the system suffers data loss and storage leaks when the reference database becomes corrupted
Solution Approach 1:
The reference database is divided into multiple independent reference databases (first reference database and second reference database), each managing references for different subsets of data containers. This segmentation ensures that corruption in one reference database does not affect the others, thereby improving reliability while maintaining manageable complexity through modular organization.
2Ease of repair
If the deduplication system is taken offline to repair the reference database, then the reference database can be fully repaired, but service level requirements are not met
Solution Approach 1:
The system enables online repair of reference databases by maintaining multiple reference databases simultaneously. When one reference database needs repair, the system continues to operate using other reference databases, ensuring continuous data access and system availability while the repair process occurs in the background without taking the entire system offline.
3Reliability
If multiple reference databases are maintained, then online repair and continuous access are enabled, but the system complexity increases
Solution Approach 1:
The system segments data containers into different subsets, with each reference database managing a specific subset. This segmentation strategy simplifies the management of multiple reference databases by creating clear boundaries and responsibilities for each database, reducing the overall complexity compared to a monolithic approach while maintaining high availability through redundancy.
Data Source
AI summary
A method for managing deduplication reference data may include (1) identifying multiple of data containers configured to store a plurality of deduplicated data segments that are referenced by multiple data objects within a deduplicated data system, (2) maintaining multiple reference databases including (i) a first reference database corresponding to a first subset of the data containers and (ii) a second reference database corresponding to a second subset of the data containers, the second subset differing from the first subset, (3) determining that a data object references at least one segment within a first data container within the first subset but does not reference any data segment within a second data container within the second subset and (4) updating the first reference database with information specifying that the data object references at least one data segment within at least one data container within the first subset of data containers.