Domain-Specific Reference Sets for Flash Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication solutions for flash storage devices face inefficiencies due to high overhead in managing global reference sets, leading to increased computing costs and reduced time efficiency, as they struggle to effectively minimize memory waste and construct optimal reference sets.
Innovation Solution
The implementation of local, namespace, domain, and synthetic reference sets, which are constructed and managed within specific storage device chunks or domains, allowing for efficient deduplication without the need for maintaining use counts and reducing overhead, while also utilizing pre-configured reference sets for improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If global reference sets are used for data deduplication in flash storage devices, then deduplication coverage is improved, but overhead and computing costs increase significantly
Solution Approach 1:
The patent divides the global reference set into multiple domain-specific reference sets based on data characteristics and access patterns. Each domain reference set manages a specific subset of data blocks, reducing the overhead of managing a single large global reference set while maintaining comprehensive deduplication coverage across different data domains.
Solution Approach 2:
The patent implements domain-specific reference sets that are tailored to local data characteristics within specific domains. Each domain reference set is optimized for its particular data type and access pattern, improving deduplication efficiency for local data while reducing the computational overhead compared to a uniform global reference set approach.
2Quantity of substance
If the concentration of flash storage device increases with more data blocks, then storage capacity is improved, but overhead increases and performance decreases
Solution Approach 1:
The patent segments the large concentrated data blocks into domain-specific subsets managed by separate reference sets. This segmentation allows parallel processing of deduplication operations across different domains, improving write and rewrite performance while maintaining high storage capacity utilization.
Solution Approach 2:
The patent implements dynamic domain reference sets that adapt to changing data patterns and access behaviors. As data concentration increases, the system dynamically adjusts domain boundaries and reference set compositions to optimize performance, preventing the overhead from linearly increasing with storage capacity.
3Reliability
If traditional reference set management is used, then deduplication functionality is provided, but memory waste occurs and reference set construction is inefficient
Solution Approach 1:
The patent changes the organizational parameters of reference sets from a single global structure to multiple domain-specific structures with different characteristics. This parameter change enables more efficient memory utilization by allocating reference set entries based on actual domain data patterns, reducing memory waste while maintaining comprehensive deduplication functionality.
Data Source
AI summary
By way of example, a data storage system may comprise, a non-transitory storage device storing data blocks in chunks, and a storage logic coupled to the non-transitory storage device that manages storage of data on the storage device. The storage logic is executable to receive a data stream for storage in a non-transitory storage device, the data stream including one or more data blocks, analyze the data stream to determine a domain, retrieve a pre-configured reference set based on the domain, and deduplicate the one or more data blocks of the data stream using the pre-configured reference set.