Block Storage Deduplication with Retention Date Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage systems face challenges in managing data retention and deduplication simultaneously, as they often require different retention dates for data blocks, leading to conflicting outcomes and inefficiencies in maintaining single copies of blocks with varying retention dates.
Innovation Solution
A storage system that assigns retention dates to client data at the block level, ensuring only a single copy of each block is maintained, even if duplicates have different retention dates, and automatically deletes blocks when all instances have reached their retention date, regardless of client availability or deduplication logic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is performed at block level, then storage efficiency is improved, but it becomes difficult to maintain different retention dates for different copies of the same data block
Solution Approach 1:
The patent segments the retention date management by introducing a reference count mechanism that tracks the number of active references to each deduplicated block. This allows the system to maintain a single copy of each block while managing retention policies through reference counting, where the block is retained as long as any reference is active and can be deleted when all references expire.
Solution Approach 2:
The patent introduces an intermediary reference count mechanism that mediates between the deduplication logic and retention date requirements. This reference count acts as a mediator that allows multiple retention date requirements to be satisfied through a single block copy, resolving the conflict between deduplication and differentiated retention policies.
2Reliability
If multiple copies of the same data block are retained with different retention dates, then retention policy compliance is improved, but storage requirements increase
Solution Approach 1:
The patent uses copying in the form of references rather than physical copies. Instead of creating multiple physical copies of data blocks with different retention dates, the system creates multiple reference entries that point to the same physical block. This allows the system to comply with different retention policies through reference management while maintaining only a single physical copy of each block.
Solution Approach 2:
The patent implements a reference counting mechanism that automatically discards (deletes) data blocks when their retention period expires and all references to them are gone. This allows the system to maintain blocks only as long as needed while efficiently recovering storage space by deleting blocks that are no longer required by any retention policy.
3Quantity of substance
If a single copy of each data block is maintained through deduplication, then storage efficiency is improved, but determining when to delete the block becomes complex when multiple clients have different retention dates
Solution Approach 1:
The patent implements a reference counting feedback mechanism that automatically tracks when blocks can be deleted. Each time a reference to a block is created or removed, the reference count is updated, providing feedback that determines whether the block should be retained or deleted. This automated feedback system simplifies the deletion logic by removing the need for complex manual determination of when blocks should be deleted.
Solution Approach 2:
The patent makes the block deletion process self-service through automated reference counting. The system automatically determines when blocks can be deleted by monitoring reference counts, eliminating the need for complex external management logic. The block deletion is self-managed through the reference counting mechanism that automatically identifies when all retention requirements have been satisfied.
Data Source
AI summary
One example method includes maintaining a deduplication data structure including one or more entries that each identify a respective fingerprint, and pointer, and also maintaining a ClientBlockList data structure comprising one or more entries that each identify a respective handle, retention date, and block, receiving a write request that identifies a handle, retention date, and block, computing a fingerprint of the block identified in the write request, determining, by the server, whether the fingerprint is in the deduplication data structure, and when the fingerprint is not in the deduplication data structure, storing the block identified in the write request at location ‘L’ in the deduplication data structure, and adding, to the deduplication data structure, an entry that identifies the fingerprint and the location ‘L,’ and adding, to the ClientBlockList data structure, an entry that identifies the handle, retention date, and fingerprint.