Coarse-Fine Reference List Management for Scalable Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication-based storage systems face inefficiencies in managing large numbers of segment objects due to limitations in reference counting and traditional reference lists, which can lead to incorrect resource reclamation and performance issues as the system scales.
Innovation Solution
Implementing a reference list management system with coarse and fine level entries, where coarse level entries are updated until a threshold is reached, then switching to fine level management, and associating entries with backup transactions to reduce processing time and overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If reference counting is used to track segment object usage, then resource reclamation can be automated, but the system becomes vulnerable to incorrect updates that may lead to premature deletion or failure to reclaim segments
Solution Approach 1:
The patent introduces reference list entries as an intermediary data structure between files and segment objects. Instead of directly counting references, the system maintains explicit list entries that record which files reference which segments. This intermediary approach allows for more reliable tracking because each reference is explicitly recorded rather than implicitly counted, reducing the risk of incorrect updates while still enabling automated reclamation when segments are no longer referenced.
2Measurement precision
If traditional fine-level reference lists are maintained for each segment object, then accurate tracking of file usage is possible, but processing time and system overhead increase significantly as the number of segment objects grows to billions
Solution Approach 1:
The patent segments the reference list management into two levels: container-level (coarse) and segment-level (fine). At the container level, the system maintains a reference list of files that reference segments within each container. At the segment level, it maintains references from containers to segments. This segmentation allows the system to track usage accurately while reducing processing overhead by operating primarily at the coarser container level, only diving into fine-grained segment management when necessary.
3Quantity of substance
If the system scales to store hundreds of terabytes of data with billions of segment objects, then storage capacity increases, but the management overhead and indexing complexity become unacceptable
Solution Approach 1:
The patent changes the dimension of reference list management by organizing references through containers as an intermediate layer. Instead of directly managing file-to-segment references across the entire system, the system introduces container-level abstraction. Files reference containers, and containers reference segments, creating a hierarchical structure that reduces the complexity of direct file-segment management while enabling the system to scale to billions of segment objects.
4Reliability
If reference lists are updated at every file deletion to ensure accurate reclamation, then resource reclamation accuracy is maintained, but system performance deteriorates due to frequent updates on large datasets
Solution Approach 1:
The patent applies partial action by updating reference lists selectively rather than comprehensively at every file deletion. The system updates container-level reference lists when files are deleted, and only updates segment-level references when necessary (e.g., when a container is fully emptied or when segment-level tracking is required). This partial updating approach maintains sufficient accuracy for resource reclamation while significantly reducing the performance impact compared to updating every reference at every deletion.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system and method for managing a resource reclamation reference list at a coarse level. A storage device is configured to store a plurality of storage objects in a plurality of storage containers, each of said storage containers being configured to store a plurality of said storage objects. A storage container reference list is maintained, wherein for each of the storage containers the storage container reference list identifies which files of a plurality of files reference a storage object within a given storage container. In response to detecting deletion of a given file that references an object within a particular storage container of the storage containers, a server is configured to update the storage container reference list by removing from the storage container reference list an identification of the given file. A reference list associating segment objects with files that reference those segment objects may not be updated response to the deletion.