SSD Controller Local Deduplication Hash Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems lack efficient deduplication capabilities at the storage media level, leading to unnecessary data duplication and reduced storage capacity, as they rely on server-level analysis without intelligent storage media to determine data redundancy.
Innovation Solution
Implementing a data storage device with a controller that checks for duplicate data and stores a pointer instead of the actual data if a copy is already present, using local deduplication logic within the storage device, thereby reducing storage needs and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in storage media without deduplication, then storage capacity is consumed by duplicate data, but storage simplicity is maintained
Solution Approach 1:
The storage media device performs deduplication operations autonomously using its own controller and processing capabilities, without requiring external server-level intervention. The controller computes hashes of incoming data blocks, queries local hash tables for duplicates, and stores pointers instead of duplicate data, enabling the storage device to serve its own deduplication needs independently
Solution Approach 2:
A hash table data structure is introduced as an intermediary mechanism between incoming data and storage media. The hash table stores hash values of data blocks as keys, enabling efficient O(1) lookup to detect duplicates. This intermediary structure facilitates rapid duplicate detection without requiring full data comparison, resolving the contradiction between deduplication effectiveness and system complexity
2Productivity
If server-level analysis is used for deduplication, then centralized control is achieved, but storage performance is reduced due to additional processing overhead
Solution Approach 1:
The deduplication function is segmented from the server-level storage system and embedded directly into the storage media device controller. This segmentation moves the computational workload from the server to the storage device, enabling parallel processing where the server continues to manage high-level operations while the storage device handles local deduplication independently, thereby improving overall storage write performance without concentrating all processing overhead at the server level
Solution Approach 2:
The storage device performs deduplication checks on incoming data blocks before they are written to storage media. By computing hashes and querying the hash table in advance of the write operation, the system identifies and eliminates duplicates proactively, preventing duplicate data from occupying storage capacity in the first place. This preliminary action improves storage efficiency without requiring post-write processing
3Quantity of substance
If duplicate data is stored, then data retrieval is simplified, but storage capacity is wasted
Solution Approach 1:
Instead of storing duplicate data blocks, the system stores a pointer or reference to the original data block that already exists in storage. The hash table maintains the mapping between hash values and physical storage locations, enabling the system to retrieve data by following pointers to unique data blocks. This copying approach eliminates redundant data storage while preserving data accessibility through the pointer mechanism
Data Source
AI summary
In some aspects, devices, systems, and methods are provided that relate to data deduplication performed in data storage devices, such as solid-state drives (SSD) or drives of any other type. In some aspects, devices, systems, and methods are provided that relate to hierarchical data deduplication at a local and system level, such as in a storage system built with one or more SSDs having built-in data deduplication functionality. The hierarchical data deduplication utilizes the IDs in the data storage devices to decide if the incoming data has to be stored or if a copy of the incoming data is already stored. In hierarchical data deduplication, no IDs (or signatures) are required to be stored at a system level. In some aspects, data steering is provided that enables data storing coordination in a system that consists of a set of data storage device (e.g., SSDs) having built-in data deduplication.