Inline Deduplication Pipeline With Persistent Memory Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication systems inefficiently compare large data file segments against previously stored segments, wasting resources and causing high latency in data processing.
Innovation Solution
Employing a low-latency hardware accelerator and persistent memory to compress and log unique data segments directly into persistent memory, reducing the need for intermediate copies and minimizing latency in the deduplication process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data file segments are directly compared against previously stored segments, then deduplication accuracy is improved, but system resource consumption increases and processing latency increases
Solution Approach 1:
The patent segments data into fixed-size data file segments and generates fingerprints for each segment. This segmentation allows the system to compare small fingerprint values (e.g., 20-byte SHA-1 hashes) instead of large data segments, reducing computational resources while maintaining deduplication accuracy through hierarchical comparison.
Solution Approach 2:
The patent introduces fingerprints as an intermediary representation between the original data segments and the comparison process. By comparing fingerprints rather than raw data segments, the system achieves accurate duplicate detection with significantly reduced resource consumption, as fingerprints are small fixed-size values that can be efficiently stored and compared.
2Measurement precision
If data file segments are directly compared against previously stored segments, then deduplication accuracy is improved, but processing speed decreases
Solution Approach 1:
The patent divides data into segments and computes fingerprints for each segment, enabling parallel processing of multiple segments simultaneously. This segmentation approach maintains deduplication accuracy while improving processing speed through efficient utilization of system resources and reduced comparison complexity.
Solution Approach 2:
The patent creates fingerprint copies of data segments that can be stored in memory for rapid comparison. By working with these compact fingerprint representations rather than original large segments, the system achieves both accurate duplicate detection and high processing throughput, as fingerprints can be quickly compared against stored fingerprint indexes.
3Quantity of substance
If unique data segments are compressed and logged into persistent memory, then storage capacity efficiency is improved, but latency increases
Solution Approach 1:
The patent performs compression of unique data segments immediately upon identification, before writing to persistent memory. This preliminary compression action reduces the amount of data that needs to be persisted, improving storage capacity efficiency. The system is designed to handle this compression quickly using hardware accelerators, minimizing the latency impact.
Solution Approach 2:
The patent uses persistent memory as an intermediary storage layer between the compression process and final storage. This persistent memory layer allows for fast writing of compressed data segments, reducing the overall latency. The system architecture separates the compression function from the persistence function, enabling parallel operations and reducing waiting time.
4Adaptability or versatility
If intermediate copies of data segments are created during processing, then processing flexibility is improved, but memory usage increases and latency increases
Solution Approach 1:
The patent extracts only the essential fingerprint information from data segments for storage and comparison purposes, rather than creating and maintaining complete intermediate copies of the original data. This extraction approach provides sufficient processing flexibility for deduplication operations while dramatically reducing memory usage, as fingerprints are small fixed-size values that require minimal storage space.
Data Source
AI summary
A processor that executes an inline deduplication pipeline identifies any data segment which is created from data received with a write request from a client, and which is unique relative to data segments stored by the inline deduplication pipeline. The processor logs any segment reference of any data segment which is identified as unique, and any data received from the client and yet to be anchored into any data segment, into persistent memory, which has a lower write latency than any non-volatile random-access memory device that is external to the persistent memory. A hardware accelerator, which has a lower compression latency than the processor, compresses any data segment identified as unique, and stores any compressed data segments into the persistent memory. The client is enabled to provide additional data by the processor acknowledging to the client that the data received from the client is stable in the persistent memory.