Key-Value Deduplication Using Hash-to-Pointer Indirection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional key-value/object-storage systems contain significant redundant information, leading to inefficient use of storage space due to duplicate data blocks.
Innovation Solution
A key-value deduplication system that uses a hash-to-pointer technique to identify and manage unique data blocks, incrementing reference counts and avoiding storage of duplicates, while maintaining indirection between a host device and storage device using a unique data block indirection table.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional key-value storage systems store all data blocks, then data retrieval is straightforward, but storage space is wasted due to redundant duplicate data blocks
Solution Approach 1:
The patent segments data storage into two distinct components: a compact key-value metadata structure that stores only unique data block references and access information, and a separate data block storage area. This segmentation eliminates redundant storage of duplicate data blocks while maintaining efficient access paths through the metadata structure, thereby improving storage space utilization without excessively increasing overall system complexity.
Solution Approach 2:
The patent introduces a key-value metadata structure as an intermediary layer between the host system and the actual data blocks. This metadata structure contains references to data blocks and access information, acting as a mediator that enables efficient data retrieval without requiring the host to directly manage raw data blocks. The intermediary handles the complexity of deduplication and reference tracking, simplifying the overall data management process.
2Quantity of substance
If duplicate data blocks are eliminated through deduplication, then storage capacity is optimized, but system complexity increases due to additional management structures
Solution Approach 1:
The patent creates a compact metadata copy that contains essential access information and references to data blocks, rather than storing complete data block copies. This metadata copying approach enables the system to track and manage unique data blocks efficiently while avoiding the storage overhead of duplicating actual data content, thus optimizing storage capacity with controlled complexity increase.
Solution Approach 2:
The patent changes the representation parameters of data storage by transitioning from storing complete data block content to storing compact metadata with references. This parameter change involves encoding data block identifiers, access counts, and metadata information in a optimized format that reduces the complexity burden while maintaining the ability to efficiently manage deduplicated storage capacity.
Data Source
AI summary
One embodiment provides an apparatus. The apparatus include a device storage logic. The device storage logic is to determine a key-based pointer based, at least in part, on a key included in an input key-value (KV) pair received from a host device and to determine whether a unique input KV data block included in the input KV pair is duplicated in a nonvolatile memory circuitry of a storage device. The device storage logic is further to set a descriptor pointer associated with the unique input KV data block to point to a physical nonvolatile memory (NVM) address associated with an existing unique data block and increment a first reference count associated with the existing unique data block, if the unique input KV data block is a duplicate of the existing unique data block, or store the input KV data block to a physical NVM location associated with a selected physical NVM address, set the descriptor pointer to point to the selected physical NVM address and set a second reference count associated with the selected physical NVM address to one, if the unique input KV data block is not duplicated in the NVM circuitry.