KVSSD Deduplication via Internal Controller Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication techniques in solid-state drives face limitations such as memory preallocation, high CPU usage for hash generation, and increased latency due to multiple I/O operations, especially when determining duplicate data blocks.
Innovation Solution
A key-value solid-state drive (KVSSD) system with separate containers for dedup keys and user data, where dedup keys are generated internally and used to determine duplicate blocks, reducing the need for external CPU processing and optimizing I/O operations by storing metadata to track references.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a preallocated memory dictionary is used for deduplication, then lookup operations can be performed, but the dictionary size is limited by reserved memory and cannot grow with the data set
Solution Approach 1:
The patent implements a dynamic dictionary that can grow and shrink based on memory availability and deduplication needs. The dictionary is no longer static with preallocated size, but dynamically adjusts its capacity to match the growing data set while managing memory resources efficiently through allocation and deallocation operations.
Solution Approach 2:
The system changes the parameter of dictionary capacity from a fixed preallocated value to a dynamic value that can be adjusted based on system conditions. This allows the dictionary size to adapt to varying data sets and memory availability, resolving the contradiction between having sufficient storage capacity and managing limited memory resources.
2Extent of automation
If flash memory is used for data deduplication with external CPU hashing, then deduplication can be implemented, but multiple I/O operations are required increasing latency
Solution Approach 1:
The patent merges the hashing function and deduplication logic directly into the SSD controller, combining previously separate functions (CPU-based hashing and flash memory storage) into a unified integrated system. This eliminates the need for separate I/O operations to transfer data between CPU and flash memory, reducing latency while maintaining full deduplication functionality.
Solution Approach 2:
The SSD controller acts as an intermediary that performs hashing operations internally before data is written to flash memory. This intermediate hashing step within the controller eliminates the need for external CPU involvement and multiple I/O operations, directly reducing the time loss associated with data transfer and processing.
3Ease of operation
If external CPU is used for hash generation, then deduplication logic can be implemented, but additional I/O operations and latency are introduced
Solution Approach 1:
The SSD controller performs hashing operations on its own without requiring external CPU assistance. The controller is self-sufficient in generating hashes and managing deduplication logic internally, eliminating the need for complex interactions with external processors and reducing the number of I/O operations required.
Data Source
AI summary
Deduplication functionality is provided for a key-value solid-state drive (KVSSD) that includes a storage space and a controller. The storage space contains first and second containers. The first container includes a first virtual space and stores a key and at least one dedup key associated with the key. Each dedup key corresponds to a block of user data, and each block includes a predetermined size. The second container includes a second virtual space and stores each dedup key, the block of user data associated with the dedup key and metadata associated with the block of user data. The controller determines whether a block of user data received by the KVSSD is a duplicate block of data stored in the second container by determining whether a dedup key for the received block of user data matches a stored dedup key in the first container.