Deduplicated Data Garbage Collection via Grouped Reference Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplicated data systems face inefficiencies in managing and removing unreferenced data segments, as they require time-consuming and resource-intensive processes like bilateral referencing and mark-and-sweep systems to ensure data segments are not prematurely deleted.
Innovation Solution
The system divides data objects into groups and focuses mark-and-sweep operations on changed groups, allowing for efficient identification and removal of unreferenced data segments by monitoring changes and creating maps of referenced segments, thereby reducing the computational resources needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional bilateral referencing systems are used to ensure data segments are not prematurely removed, then data reliability is improved, but system complexity and resource consumption increase
Solution Approach 1:
The patent extracts the referential list management from the data segment structure and consolidates it into a centralized metadata structure. Instead of each data segment maintaining its own referential list, the system uses a unified metadata repository that tracks all references centrally, reducing complexity while maintaining reliability
Solution Approach 2:
The patent introduces a metadata structure as an intermediary between files and data segments. This metadata layer acts as a mediator that tracks references without requiring direct bidirectional linkages between files and segments, simplifying the reference management mechanism while ensuring data integrity
2Reliability
If mark-and-sweep systems check each data segment to ensure it is not referenced before deletion, then data reliability is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent performs preliminary actions by maintaining up-to-date reference count information in the metadata structure during normal file operations. When a file is deleted or modified, the metadata is immediately updated to reflect changes in references, so that garbage collection can proceed without performing exhaustive checks, significantly reducing processing time while maintaining deletion safety
Solution Approach 2:
The system implements continuous feedback by updating metadata reference information in real-time as files are created, modified, or deleted. This feedback mechanism ensures the metadata always accurately reflects current reference states, enabling efficient and safe garbage collection without exhaustive scanning
3Stability of the object's composition
If bilateral referencing systems update both file and data segment referential lists when a file is updated, then data consistency is improved, but processing overhead increases
Solution Approach 1:
The patent extracts the reference tracking function from the data segment structure and places it in a centralized metadata structure. When files are updated, only the metadata needs to be updated rather than maintaining and synchronizing multiple referential lists across files and segments, reducing computational overhead while preserving consistency
Solution Approach 2:
The metadata structure serves multiple functions: it tracks references for garbage collection, maintains data consistency during file operations, and provides a unified view of all data segment references. This multi-functional approach eliminates the need for separate referential list management in both files and segments
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method for removing unreferenced data segments from deduplicated data systems may include: 1) identifying a deduplicated data system that contains a plurality of data objects, 2) dividing the data objects within the deduplicated data system into a plurality of data object groups, 3) identifying, within the data object groups, at least one data object group that has changed subsequent to a prior garbage-collection operation that removed data segments that were not referenced by data objects within the deduplicated data system, 4) identifying at least one container within the deduplicated data system that contains data segments referenced by data objects within the changed data object group, and then, for each identified container, 5) removing data segments from the identified container that are not referenced by data objects within the deduplicated data system. Various other methods, systems, and computer-readable media are also disclosed.