Multiple Fingerprint Tables for Flash Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face challenges in efficiently removing duplicate data blocks at a fine granularity from storage arrays and subsequently accessing them efficiently, leading to performance issues and storage inefficiencies.
Innovation Solution
The implementation of a deduplication system that uses fingerprint tables and secondary indexes to efficiently identify and remove duplicate data blocks at a fine granularity, while also optimizing access times through strategic storage and retrieval of metadata.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If a single large fingerprint table is used for deduplication, then storage efficiency is improved through comprehensive duplicate detection, but access time and system performance deteriorate due to the large size of the table
Solution Approach 1:
The patent divides the large fingerprint table into multiple smaller fingerprint tables (first fingerprint table, second fingerprint table, etc.). Each table stores fingerprints for different portions of the storage array. This segmentation reduces the access time for any single table while maintaining comprehensive deduplication coverage across the entire storage system.
Solution Approach 2:
The patent introduces a hierarchical dimension to the fingerprint table structure by creating multiple levels or types of fingerprint tables. The system can search through different tables based on the specific data portion being accessed, adding a dimensional organization that improves access efficiency without sacrificing deduplication completeness.
2Loss of substance
If fine granularity deduplication is implemented, then storage efficiency is improved by removing more duplicate blocks, but device complexity increases due to the need for multiple fingerprint tables and indexes
Solution Approach 1:
The storage array is divided into multiple portions, each associated with specific fingerprint tables. This segmentation allows fine-grained deduplication to be applied locally to each portion, reducing the overall complexity compared to managing a single large-scale fine-grained deduplication system for the entire array.
Solution Approach 2:
Different fingerprint tables are optimized for different portions of the storage array, allowing each table to be tailored to its specific data characteristics. This local optimization achieves fine-grained deduplication where needed while avoiding unnecessary complexity in other areas.
3Speed
If multiple fingerprint tables are used for deduplication, then access performance is improved through smaller table sizes, but the amount of metadata and storage overhead increases
Solution Approach 1:
The metadata (fingerprint tables) is segmented into multiple smaller tables corresponding to different storage array portions. This segmentation reduces the amount of metadata that needs to be loaded and searched at any given time, improving access performance while the total metadata quantity remains proportional to the data size.
Solution Approach 2:
The system maintains fingerprint tables for only the portions of data that are currently active or frequently accessed, rather than maintaining comprehensive metadata for all possible data. This partial action approach reduces metadata overhead while maintaining adequate access performance for active data.
Data Source
AI summary
An entry for data to be stored in one or more storage devices of a storage system is added by a storage system controller to a particular fingerprint table of multiple fingerprint tables having a range of probabilities that a probability of the data being deduplicated falls within. One of the multiple fingerprint tables is modified in response to an occurrence of an event.