Coordinated Inline Deduplication in Storage Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face challenges in efficiently deduplicating data across nodes while flushing data to storage, leading to increased memory requirements and inconsistent deduplication performance.
Innovation Solution
A method for coordinated inline deduplication among nodes, where a node stores data and fingerprints in a page descriptor ring, identifies entries with specific parity, deduplicates them using a fingerprint database, synchronizes with a peer node, and replaces entries with deduplication results, ensuring efficient deduplication during flushing to storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in multiple folders across nodes without coordinated deduplication, then data redundancy is reduced, but memory requirements increase and deduplication performance becomes inconsistent
Solution Approach 1:
The system divides the deduplication task into segments by assigning different fingerprint parities to different nodes. Each node only processes entries with its assigned parity, reducing the memory footprint and processing load on each individual node while maintaining system-wide deduplication effectiveness.
Solution Approach 2:
Multiple nodes work together to handle deduplication for different fingerprint parities. By combining the capabilities of multiple nodes, the system achieves comprehensive deduplication coverage without requiring any single node to manage all data, thus reducing individual memory requirements.
2Reliability
If all nodes perform complete deduplication independently, then deduplication coverage is maximized, but flushing performance decreases due to memory constraints
Solution Approach 1:
The deduplication workload is segmented across multiple nodes based on fingerprint parity. Each node handles only its assigned parity, reducing the volume of data each node must process and flush, thereby improving flushing performance while maintaining comprehensive coverage through coordinated operation.
Solution Approach 2:
Nodes communicate fingerprint information and deduplication results as intermediaries. When a node encounters a fingerprint it doesn't own, it queries peer nodes for the relevant information, allowing it to proceed with flushing without waiting for complete deduplication data, thus improving performance.
3Measurement precision
If nodes store complete fingerprint databases locally, then deduplication accuracy is improved, but memory consumption increases
Solution Approach 1:
The fingerprint database is segmented by parity and distributed across different nodes. Each node stores only the fingerprints with its assigned parity, reducing local memory consumption while maintaining the ability to accurately identify and deduplicate all data types through coordinated access to peer nodes.
Data Source
AI summary
Techniques described herein coordinate inline deduplication among nodes in a storage system. The method includes storing, in a page descriptor ring on a node, data and a fingerprint associated with the data in an entry. The method includes determining that a flushing work set (FWS) has been frozen. The node identifies, in the page descriptor ring, entries associated with the frozen FWS and having fingerprints with a parity associated with the node. The node deduplicates the entries based on a fingerprint database on the node. The node synchronizes deduplication of the frozen FWS with a peer node, so as to receive deduplication results concerning entries having fingerprints with a parity associated with the peer node. The node replaces entries in the page descriptor ring with the deduplication results from the peer node, and flushes entries in the frozen FWS to a storage device.