Deduplicated Storage Expired Data Detection via Reference Counting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional deduplicated storage systems face inefficiencies in detecting and removing expired data objects due to time-consuming mark-and-sweep techniques, leading to prolonged retention of unnecessary data.
Innovation Solution
A system and method that utilize unique identifiers associated with data objects, where the presence of these identifiers in a list determines if a data object is not expired, allowing for efficient detection and automatic deletion of expired data objects upon request, eliminating the need for extensive mark-and-sweep processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If mark-and-sweep techniques are used to detect expired data objects, then the system can identify and remove unnecessary data, but the process is time-consuming and expired data objects remain in the data store for considerable periods between sweeps
Solution Approach 1:
The patent applies preliminary action by maintaining a reference count for each data object that is updated in real-time whenever files are inserted or removed. This pre-computed reference count immediately indicates whether a data object is expired, eliminating the need for time-consuming periodic mark-and-sweep traversal operations. The system performs the detection work incrementally as operations occur, rather than batching it into lengthy periodic sweeps.
2Reliability
If mark-and-sweep techniques are used to detect expired data objects, then the system can identify and remove unnecessary data, but expired data objects remain in the data store for considerable periods between sweeps
Solution Approach 1:
The patent applies self-service by making each data object self-describing through its reference count attribute. The data object inherently carries the information needed to determine its own expiration status, eliminating the need for external mark-and-sweep processes to traverse and analyze metadata. When the last file referencing a data object is removed, the reference count automatically becomes zero, immediately标识ing the object as expired for immediate deletion.
3Loss of information
If traditional metadata tracking is used, then the system can maintain relationships between files and data objects, but the detection process becomes time-consuming
Solution Approach 1:
The patent extracts the essential expiration detection information (reference count) from the complex metadata structure and makes it directly accessible as an attribute of each data object. Instead of traversing entire metadata structures to determine expiration status, the system directly reads the pre-maintained reference count, significantly reducing the time and computational resources needed for expiration detection while preserving complete tracking of file-data object relationships.
Data Source
AI summary
A system, method, and medium for detecting expired data in a data deduplicated storage system are disclosed. The system receives a request to insert a file into the deduplicated storage system. In response, the system determines a unique identifier for the file and stores the unique identifier in a list of unique identifiers associated with a data object that stores the file content. The system determines that the data object is not expired based at least in part on the identifier being present in the list of unique identifiers.