Storage Device Deduplication Manager Host Load Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems waste capacity due to duplicate data storage, as they store multiple copies of the same data, leading to increased computational load and performance bottlenecks in the host device during deduplication operations.
Innovation Solution
Implementing a deduplication manager within storage devices that communicates peer-to-peer to identify and manage duplicate data, allowing only one original data set to be stored while referencing duplicates, thereby reducing storage waste and computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If duplicate data are stored multiple times in the storage device, then data redundancy and backup are achieved, but storage capacity is unnecessarily wasted
Solution Approach 1:
Instead of storing actual duplicate data copies, the patent creates a mapping relationship where duplicate data references point to the original data. The mapping table stores mappings between duplicate data identifiers and original data locations, allowing multiple logical copies to reference a single physical storage location, thus eliminating redundant storage while maintaining data availability for backup purposes
2Quantity of substance
If deduplication operations are performed to reduce storage waste, then storage capacity efficiency is improved, but computational load and performance bottlenecks increase in the host device
Solution Approach 1:
The patent extracts the deduplication computational tasks from the host device and relocates them to the storage device itself. The storage device autonomously performs fingerprint generation, duplicate detection, and mapping table updates by processing data locally during write operations, eliminating the need for intensive host-side computation while maintaining storage capacity efficiency
Solution Approach 2:
The storage device performs self-managed deduplication by generating fingerprints for incoming data, comparing them against existing mappings, and automatically creating or updating references in the mapping table. This self-service mechanism eliminates the need for host device intervention in computational tasks, reducing host computational load while achieving efficient storage capacity utilization
3Ease of operation
If deduplication operations are performed by the host device, then duplicate data management is achieved, but I/O load and processing time increase
Solution Approach 1:
The patent extracts deduplication processing from the host device's I/O path and implements it within the storage device. The storage device generates fingerprints, queries the mapping table, and creates references locally during data write operations, eliminating the need for host-device communication loops and reducing I/O load while maintaining ease of duplicate data management
Solution Approach 2:
The patent performs deduplication actions preliminarily during the data write process itself. By generating fingerprints and creating mapping entries at the time of data ingestion, the system avoids subsequent processing delays and I/O operations that would occur if deduplication were performed as a separate post-processing step, thus reducing overall processing time while maintaining operational simplicity
Data Source
AI summary
Disclosed is a storage device which communicates with a first external storage device and includes a deduplication manager. A method of operating the storage device includes broadcasting, by the deduplication manager, a first fingerprint of first data, receiving a first response corresponding to the first fingerprint from the first external storage device, determining, by the deduplication manager, whether a reference flag of the first response has a first value or a second value, updating, by the deduplication manager, first mapping information of a mapping table based on the first response, based on a determination that the reference flag of the first response has the first value, and performing, by the deduplication manager, deduplication of the first data, based on the first response, based on a determination that the reference flag of the first response has the first value.