In-line Deduplication in Distributed Storage via Fingerprinting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in read and write operations due to the manner in which data is managed and processed, particularly in handling large data volumes and the need for deduplication, error detection, and data protection, which can impact performance and latency.
Innovation Solution
A system and method that involve a client application node generating a fingerprint for data, issuing I/O requests to a storage pool based on a data layout, and transmitting the fingerprint to a metadata node for deduplication, while also implementing in-line deduplication, error detection, and data protection mechanisms to enhance storage efficiency and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored using traditional storage systems with centralized metadata management, then data protection and deduplication are achieved, but write and read operation latency increases due to additional processing steps
Solution Approach 1:
The patent segments the storage system into independent storage nodes that perform deduplication and data protection operations locally without centralized coordination. Each storage node maintains its own metadata and processes I/O requests autonomously, eliminating the bottleneck of centralized metadata management while preserving data protection and deduplication capabilities.
Solution Approach 2:
The patent implements preliminary action by having storage nodes pre-compute fingerprints for data blocks and maintain local metadata structures in advance. When I/O requests arrive, the nodes can immediately check for duplicates and apply protection mechanisms using pre-computed information, avoiding real-time processing delays.
2Productivity
If in-line deduplication is implemented in distributed storage systems, then storage efficiency improves, but system complexity increases due to fingerprint generation and metadata management
Solution Approach 1:
The patent applies self-service by enabling storage nodes to autonomously generate fingerprints, detect duplicates, and manage their own metadata without external coordination. Each node independently performs deduplication operations on incoming data streams, eliminating the need for complex centralized deduplication management while achieving high storage efficiency.
Solution Approach 2:
The patent makes storage nodes universal by combining multiple functions within each node: data reception, fingerprint generation, duplicate detection, metadata management, and I/O processing. This multi-functionality eliminates the need for separate specialized components, reducing overall system complexity while maintaining high deduplication efficiency.
3Speed
If direct I/O operations are performed by client application nodes to storage pool, then operation speed improves, but data consistency and protection mechanisms become more difficult to maintain
Solution Approach 1:
The patent extracts data protection and consistency management functions from the client application nodes and places them directly in the storage nodes. This allows clients to perform simple, fast direct I/O operations while the storage nodes autonomously handle fingerprint generation, duplicate detection, and protection mechanisms, maintaining data consistency without burdening the client.
Solution Approach 2:
The patent introduces storage nodes as intermediaries between client application nodes and the storage pool. These intermediaries receive I/O requests from clients, perform deduplication and protection operations locally, and then interact with the storage pool. This intermediary layer enables fast direct client-to-storage operations while maintaining data protection through local processing at the storage node level.
Data Source
AI summary
A method for deduplicating data comprising: obtaining, from a metadata node and by file system client executing on a client application node, a data layout; generating, by the client application node, a fingerprint for the data stored on the client application node; generating, by a memory hypervisor module executing on the client application node, at least one input/output (I/O) request specifying a location in a storage pool, wherein the location is determined using the data layout; issuing, by the memory hypervisor module, the at least one I/O request to the storage pool, wherein processing the at least one I/O request results in at least a portion of the data being stored at the location; and after issuing the at least one I/O request to the storage pool, transmitting the fingerprint to the metadata node, wherein the metadata node attempts to deduplicate the data using the fingerprint.