Storage Controller Data Deduplication via Chunk Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication techniques in storage networks and systems are inefficient due to outdated reference blocks, improper deletion of useful blocks, and failure to recognize slight deviations in data, leading to unnecessary storage of redundant data copies, which consumes valuable storage resources and increases costs.
Innovation Solution
A system comprising a non-volatile cache memory device and a storage controller that uses chunk reference signatures to identify matches, performs deduplication techniques based on comparison blocks, and employs a multi-layer mechanism for garbage collection to efficiently manage and promote candidate comparison blocks, thereby reducing redundant data storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exact match algorithms are used to identify redundant data, then the number of exactly identical copies is reduced, but substantially similar data with slight deviations are not recognized and stored as new data, leading to unnecessary storage consumption
Solution Approach 1:
The patent changes the matching parameter from exact byte-by-byte comparison to similarity-based comparison using hash algorithms. Instead of requiring 100% identical data, the system computes hash values and compares them to identify substantially similar data blocks, allowing for slight deviations while still recognizing redundancy. This resolves the contradiction by improving storage efficiency without sacrificing matching accuracy for similar data.
2Quantity of substance
If reference blocks are stored in limited storage space areas, then more reference blocks can be maintained, but outdated reference blocks consume valuable space that prevents updated reference blocks from being stored, reducing deduplication effectiveness
Solution Approach 1:
The patent implements a reference block management mechanism that automatically discards outdated reference blocks when storage space is limited. The system monitors the age and usage of reference blocks, and when the storage area is full, it removes older reference blocks to make space for newly updated ones. This ensures that the limited storage space is always occupied by the most relevant reference blocks, maintaining deduplication accuracy while maximizing the number of active reference blocks.
3Productivity
If reference blocks are deleted after a predetermined time or hit count, then storage space is freed for other reference blocks, but useful reference blocks may be deleted prematurely before their full usefulness is attained
Solution Approach 1:
The patent employs a feedback mechanism that continuously monitors the performance and usage patterns of reference blocks. Instead of deleting reference blocks based on fixed time or hit count thresholds, the system tracks actual deduplication effectiveness and adjusts retention policies dynamically. Reference blocks that demonstrate high usefulness are retained longer, while those with diminishing returns are removed. This feedback-driven approach ensures that storage space is optimized without prematurely deleting valuable reference blocks.
Data Source
AI summary
Various aspects for data deduplication in a storage system are provided. For instance, a storage controller may perform operations including receiving a data chunk including a set of data blocks, determining a signature for the data chunk, and comparing the signature and a set of reference signatures to determine a match. Responsive to a match, the operations may further include identifying a reference data chunk including a set of comparison blocks associated with the matched reference signature, performing a deduplication technique on the set of data blocks based on the set of comparison blocks, and identifying a subsequent reference data chunk for an estimated next data chunk based on identification of the reference data chunk and prior to receipt of the next data chunk.