Host-Assisted Deduplication in Solid-State Storage Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Solid-state data storage devices, such as those with NAND flash memory, face challenges in implementing effective data deduplication due to the absence of an in-place-update feature, leading to data storage consistency issues and increased write latency when attempting host-assisted device-level deduplication.
Innovation Solution
A host-assisted data deduplication system that uses a non-volatile memory write cache to decouple duplicate detection from the foreground write path, allowing immediate write completion acknowledgments and asynchronous duplicate detection, enabling efficient data deduplication without degrading storage consistency or performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If host-side duplicate-detection operation is directly inserted into the foreground write path, then data deduplication can be performed, but write latency increases significantly
Solution Approach 1:
The system performs duplicate detection in advance by caching data sectors in non-volatile memory before they are written to flash memory. The controller proactively checks for duplicates using the cached data and LBA information, so that when actual write operations occur, the deduplication decision is already made, eliminating the need to block the write path for detection operations.
Solution Approach 2:
The write operation is divided into two independent phases: (1) data caching phase where data sectors are stored in non-volatile memory write cache, and (2) deduplication decision phase where the controller checks for duplicates using cached data and LBAs. This segmentation allows the foreground write path to remain fast while background processes handle the compute-intensive duplicate detection.
2Productivity
If device-level data deduplication is implemented without in-place update capability, then data deduplication can be achieved, but reverse PBA-LBA mapping consistency deteriorates
Solution Approach 1:
The system performs duplicate detection and mapping decisions in advance before data is written to flash memory. The controller uses non-volatile memory to cache data sectors and their associated LBAs, then proactively determines duplicate relationships and updates mappings before the actual write completes. This ensures that the reverse PBA-LBA mapping remains consistent and up-to-date without requiring in-place updates during write operations.
Solution Approach 2:
Non-volatile memory serves as an intermediary layer between the host and flash memory. It temporarily stores data sectors and LBA information, allowing the controller to perform duplicate detection and mapping updates in a controlled manner. This intermediary enables the system to maintain accurate reverse PBA-LBA mappings while working around the limitation of flash memory's lack of in-place update capability.
Data Source
AI summary
A host-assisted data deduplication system, including: a storage device including a controller, memory, and a write cache; and a host including a data duplication module coupled to the storage device, wherein the controller of the storage device is configured to write a data sector received from the host into the write cache in the storage device and to send a write completion to the host immediately after writing the data sector to the write cache; wherein the data duplication module is configured to detect whether the data sector is identical to another data sector stored in the memory of the storage device and to asynchronously send a duplicate detection result to the controller of the storage device, and wherein the controller of the storage device is configured to perform a data deduplication process for the data sector stored in the write cache based on the duplicate detection result.