Storage Device Deduplication via Hash Table Reference Counting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication techniques in server systems require high-capacity memory, leading to increased manufacturing costs and inefficient use of storage space due to the storage of duplicate data.
Innovation Solution
A storage device with a deduplication function that uses a hash operation to identify and manage duplicate data, storing only one instance of data across multiple references by maintaining a hash-to-physical address table and updating reference counts, thereby reducing storage needs and costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If current deduplication techniques are used to eliminate duplicate data, then storage efficiency is improved, but high-capacity memory is required which increases manufacturing costs
Solution Approach 1:
The patent replaces expensive high-capacity memory with a hash table data structure that can be implemented using standard memory resources. The hash table stores only hash values and reference counts rather than actual data, dramatically reducing memory requirements while maintaining deduplication functionality.
Solution Approach 2:
The patent changes the approach from storing data in memory to storing only hash values and reference counts in a hash table. This parameter change in the data structure transforms the memory requirements from high-capacity to standard capacity, resolving the contradiction between storage efficiency and manufacturing cost.
2Ease of operation
If duplicate data is stored for each reference, then data access is simplified, but available storage space decreases
Solution Approach 1:
The patent creates a logical copy mechanism through the hash table that allows multiple references to the same physical data without duplicating the actual data storage. Each reference is tracked through the hash table entry, enabling efficient data access while preventing physical duplication.
Solution Approach 2:
The hash table serves multiple functions simultaneously: it acts as a deduplication database, a reference counter, and an address mapping structure. This multi-functionality eliminates the need for separate data structures that would otherwise be required, improving storage efficiency while maintaining ease of operation.
3Measurement precision
If hash operation is performed on all incoming data, then duplicate detection accuracy is improved, but processing time increases
Solution Approach 1:
The patent performs the hash operation immediately when data arrives, before any storage or processing decisions are made. This preliminary hashing enables immediate duplicate detection and allows the system to proceed with confident deduplication decisions without requiring additional verification steps.
Solution Approach 2:
The hash value acts as an intermediary that represents the entire data content without requiring comparison of the actual data. By using the hash value as a mediator for duplicate detection, the system achieves high accuracy without the time cost of comparing large data sets directly.
Data Source
AI summary
An operation method of a storage device includes: receiving a write request including an object identifier and data from an external device; performing a hash operation on the data to generate a hash value; determining whether an entry associated with the hash value is empty in a table; storing the data in an area of the storage device corresponding to a physical address and updating the entry to include the physical address and a reference count, when it is determined that the entry is empty; and increasing the reference count included in the entry without performing a store operation associated with the data, when it is determined that the entry is not empty, and an error message is returned to the external device when the entry associated with the hash value is not present in the table.