Lockless Deduplication via Reference Tags and Sliding Windows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In deduplicated storage systems, traditional locking mechanisms are inefficient and resource-intensive, leading to high overhead costs and increased risk of deadlocks, especially in high-scale environments with millions of data extents and concurrent workload processing.
Innovation Solution
The implementation of a lockless approach using reference tagging with sliding window time periods for data extent management, where eligibility for deletion or linkage is determined based on specific retention periods and cushions, allowing for concurrent operations without the need for locks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional locking mechanisms are used to manage data extents, then data integrity is ensured, but resource utilization increases and system performance decreases
Solution Approach 1:
The patent extracts and removes the locking mechanism from the data extent management system. Instead of using locks to ensure data integrity, the system uses reference counting and timestamp-based eligibility criteria to determine whether data extents can be safely deleted or linked, eliminating the need for serialization mechanisms while maintaining integrity.
Solution Approach 2:
The patent replaces the mechanical locking system with a reference-counting and timestamp-based decision system. Instead of using locks/mutexes to control access, the system uses software-based reference tags and time-window eligibility criteria to manage concurrent operations, substituting mechanical serialization with a different control mechanism.
2Reliability
If locking mechanisms are used to prevent deletions during linkage operations, then data integrity is maintained, but memory overhead increases
Solution Approach 1:
The patent removes the locking mechanism entirely from the system. Instead of allocating memory for locks and lock lists, the system uses reference counting and timestamp-based eligibility criteria to manage data extent operations, significantly reducing memory overhead while maintaining data integrity through alternative means.
Solution Approach 2:
The patent replaces expensive, persistent lock structures with lightweight reference tags and timestamp-based eligibility determinations. The system uses minimal metadata (reference counts and timestamps) that can be quickly computed and discarded, eliminating the need for heavy locking infrastructure.
3Reliability
If locks are used to ensure exclusive access to data extents, then data integrity is protected, but the risk of deadlocks increases
Solution Approach 1:
The patent extracts and eliminates the locking mechanism from the system, removing the source of deadlock problems. By using reference counting and timestamp-based eligibility criteria, the system achieves data integrity without the mutual exclusion and waiting queues that cause deadlocks in traditional locking systems.
Solution Approach 2:
The patent replaces the mechanical lock-based exclusion system with a reference-counting and time-window-based system. This substitution eliminates the conditions for deadlock (mutual exclusion and circular waiting) while maintaining data integrity through alternative control mechanisms.
4Reliability
If traditional serialization mechanisms are used to manage data extents, then data integrity is ensured, but processor overhead increases
Solution Approach 1:
The patent removes the serialization mechanism (locks/mutexes) from the data extent management system. Instead of using processor-intensive locking and unlocking operations, the system uses reference counting and timestamp-based eligibility determinations that require minimal processing and eliminate the need for wait queues and lock management.
Solution Approach 2:
The patent replaces the processor-intensive mechanical locking system with a software-based reference-counting and time-window system. This substitution eliminates the need for lock acquisition, release, and wait queue management, significantly reducing processor overhead while maintaining data integrity.
Data Source
AI summary
In one embodiment, a method includes receiving a query related to a first data extent. The method also includes determining whether the first data extent is eligible for deletion based on a first sliding window time period equal to a predetermined retention period added to a predetermined deletion cushion in response to the query requesting a deletion operation for the first data extent. The method also includes determining whether the first data extent is eligible for linkage based on a second sliding window time period equal to the retention period less a predetermined linkage cushion in response to the query requesting a linkage operation for the first data extent. The linkage cushion includes less time than the deletion cushion.