Multiple Deduplication Tables for Fine-Granularity Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems face inefficiencies in removing duplicate data blocks at a fine granularity, leading to suboptimal storage utilization and performance, as existing deduplication methods operate at a coarse level of granularity and can slow down storage operations.
Innovation Solution
A data storage subsystem with multiple deduplication tables that dynamically adjust based on attributes such as deduplication rates and access frequencies, employing inline and offline deduplication strategies across various storage devices, including SSDs and DRAM, to efficiently identify and remove duplicate data blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If deduplication is performed at a coarse level of granularity using traditional software applications, then storage efficiency is improved, but the ability to locate and remove duplicate data is limited and performance is reduced
Solution Approach 1:
The patent divides the storage array into multiple fine-grained data blocks and processes each block independently through the deduplication table. This segmentation allows the system to identify and remove duplicates at a much finer level than traditional coarse-grained approaches, improving both storage efficiency and performance by processing smaller units of data that can be handled more quickly and precisely.
Solution Approach 2:
The patent replaces traditional software-based deduplication mechanisms with a hardware-accelerated approach using a dedicated deduplication table integrated into the storage array. This substitution of the mechanical/software system with a specialized hardware structure enables faster processing and finer-granularity deduplication while maintaining storage efficiency.
2Quantity of substance
If deduplication operations are performed to remove duplicate data blocks, then storage utilization is improved, but storage-related operations such as reads and writes may run more slowly
Solution Approach 1:
The patent performs deduplication checks in advance using the deduplication table before actual data write operations occur. By pre-identifying duplicate blocks and their corresponding non-duplicate blocks, the system can prepare mapping information ahead of time, allowing subsequent read and write operations to proceed faster without the overhead of real-time deduplication processing.
Solution Approach 2:
The patent creates a copy of the deduplication table structure that maps virtual block addresses to physical block addresses. This copy mechanism allows the system to maintain storage utilization benefits while enabling faster operations by using the pre-computed mappings rather than performing full deduplication analysis during each read/write operation.
3Measurement precision
If multiple deduplication tables are used to store data components with different probabilities of being deduplicated, then deduplication accuracy at fine granularity is improved, but device complexity increases
Solution Approach 1:
The patent implements multiple deduplication tables with different characteristics tailored to specific data types and access patterns. Each table is optimized for particular workloads, allowing the system to achieve high deduplication accuracy for different data scenarios without requiring a completely complex unified system. The local optimization of each table reduces overall system complexity compared to a single complex table design.
Data Source
AI summary
A system and method for efficiently removing duplicate data blocks at a fine-granularity from a storage array. A data storage subsystem supports multiple deduplication tables. Table entries in one deduplication table have the highest associated probability of being deduplicated. Table entries may move from one deduplication table to another as the probabilities change. Additionally, a table entry may be evicted from all deduplication tables if a corresponding estimated probability falls below a given threshold. The probabilities are based on attributes associated with a data component and attributes associated with a virtual address corresponding to a received storage access request. A strategy for searches of the multiple deduplication tables may also be determined by the attributes associated with a given storage access request.