Deduplication Reference Counting via Metadata Journal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, concurrent updates to reference counts for deduplicated data lead to performance degradation due to the need for exclusive locks and synchronization among nodes, resulting in queued write requests and increased latency.
Innovation Solution
The use of non-exclusive locks for updating reference counts, where instructions to increment the count are stored in a metadata journal, allowing multiple requests to be processed concurrently, and caching inconsistencies are managed with flags to ensure accurate values when needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If exclusive locks are used for updating reference counts during deduplication, then data consistency is ensured, but write request latency increases and IOPS decrease due to queuing and synchronization requirements
Solution Approach 1:
A metadata journal is introduced as an intermediary component to decouple the write request processing from the reference count update operation. Write requests are logged to the journal without acquiring locks on the reference count page, allowing concurrent processing. The reference count updates are applied asynchronously from the journal entries, eliminating the need for exclusive locks during write operations while maintaining eventual consistency.
Solution Approach 2:
Write requests are pre-processed and logged to the metadata journal before the reference count updates are applied. This preliminary action allows the write operation to complete and be acknowledged to the client without waiting for the reference count to be updated, thereby reducing write latency and enabling concurrent operations on the same data pages.
2Stability of the object's composition
If exclusive locks are used for updating reference counts, then synchronization among nodes is maintained, but concurrent write request processing is blocked
Solution Approach 1:
The metadata journal serves as a mediator that captures write requests without requiring synchronization locks. Multiple nodes can write to the journal concurrently using non-exclusive locks or lock-free structures. The reference count updates are then applied in a coordinated manner from the journal entries, maintaining synchronization without blocking concurrent write submissions.
Solution Approach 2:
Write requests are captured and queued in the metadata journal as preliminary actions before synchronization and reference count updates occur. This allows the system to accept and preliminarily process write requests from multiple nodes concurrently, then apply the synchronized updates in a subsequent phase, thereby eliminating wait time for write acknowledgments.
3Productivity
If non-exclusive locks are used for metadata updates, then concurrent write requests can be processed, but cache consistency with storage may be compromised
Solution Approach 1:
The metadata journal acts as an intermediary that records all reference count updates. When cache consistency needs to be verified, the system can replay or apply the journal entries to ensure the cache reflects the current state. This allows non-exclusive locks to be used for improved concurrency while providing a mechanism to restore consistency if the cache becomes outdated.
Solution Approach 2:
The system implements feedback mechanisms where cache validity is monitored and verified against the metadata journal or storage state. When inconsistencies are detected (such as when a cached page is accessed after updates may have occurred), the cache is invalidated or reloaded from storage, ensuring consistency is restored without having used exclusive locks during the write operations.
Data Source
AI summary
A method is used for improving performance in a storage system during deduplication. A processor receives a request to write data. A determination is made that the data can be deduplicated. A non-exclusive lock for a page in storage with a reference count of the data may be obtained, and an instruction to increment the reference count for the data may be stored in a metadata journal. The non-exclusive lock for the page may be released.