Container Object Reference Counting for Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud storage systems face complexity in managing storage goals of immutability, data deduplication, retention, and expiration, particularly due to the proliferation of different objects mapping to different deduplicated extents in containers, which complicates the enforcement of retention rules and data availability.
Innovation Solution
The solution involves maintaining reference counts for chunks in container objects to track deduplication references, indicating containers as reclaim eligible when reference counts reach zero, allowing for efficient deletion and ensuring data immutability and retention rules are enforced while maintaining availability of referenced chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is implemented in WORM storage, then storage efficiency is improved, but managing retention rules and data availability becomes more complex
Solution Approach 1:
The patent introduces container objects as intermediary layers between the deduplicated chunks and the application objects. Each container object maintains reference counts for its chunks and tracks which application objects reference them. This intermediary structure allows the system to enforce retention rules at the container level while managing deduplication at the chunk level, resolving the complexity by separating these two functions into distinct management layers.
Solution Approach 2:
The system implements feedback mechanisms through reference counts that track how many application objects reference each chunk and how many containers reference each chunk. This feedback information enables the system to automatically determine when chunks can be safely deleted (when reference counts reach zero) and when containers can be reclaimed, providing automated retention management without manual intervention.
2Reliability
If reference counts are maintained for all chunks, then data availability is ensured, but storage overhead increases
Solution Approach 1:
The patent applies local quality by maintaining reference counts only where necessary - specifically in container objects that group related chunks. Not all chunks require individual reference tracking at the same level; instead, reference counts are maintained locally at the container level and selectively at the chunk level based on whether the chunk is shared across multiple containers. This localized approach ensures data availability while minimizing unnecessary storage overhead.
3Productivity
If containers are deleted when reference counts reach zero, then storage space is reclaimed, but retention rules may be violated
Solution Approach 1:
The system performs preliminary actions by marking containers as reclaim-eligible before actually deleting them. When a container's reference count reaches zero, it is marked as reclaim-eligible but not immediately deleted. This preliminary marking allows the system to verify that all retention rules are satisfied and that no application objects still reference the chunks, ensuring retention compliance before the actual deletion occurs.
Data Source
AI summary
Provided are a computer program product, system, and method for managing retained container objects in storage having referenced chunks referenced in other objects. Reference counts are maintained for referenced chunks in a container object indicating a number of references to the referenced chunks in at least one application object resulting from deduplication. The container object is indicated as reclaim eligible while the reference counts for at least one of the referenced chunks for the container object is at least one. A reference count is decremented for one of the referenced chunks in the container object in response to the at least one application object expiring. The container object, including chunks for the container object, is indicated as eligible for deletion when the reference counts for the referenced chunks in the container object are zero and the container object is indicated as reclaim eligible.