In-Line Deduplication Verification for Fingerprint Collisions

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional storage systems face inefficiencies in data management and storage operations, particularly in handling fingerprint collisions and data deduplication, which can lead to unnecessary write operations and reduced reliability.

Innovation Solution

The implementation of a storage system that uses non-volatile random access memory (NVRAM) as a buffer for write operations, offloading device management responsibilities from storage drives, and employing methods to handle fingerprint collisions through hash functions and deduplication processes to optimize storage efficiency and reliability.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If traditional storage systems perform data deduplication using hash functions, then storage efficiency is improved, but fingerprint collisions occur leading to reduced reliability

Engineering Contradiction:

Improvestorage efficiencyVSAvoiddata integrity

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The patent introduces an intermediary verification process between hash function matching and data deduplication. Before finalizing deduplication based on fingerprint matches, the system performs additional verification steps to detect and handle fingerprint collisions, thus maintaining storage efficiency while improving data integrity

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces the purely mechanical hash-based deduplication system with a more sophisticated approach that incorporates verification mechanisms. Instead of relying solely on hash function matching, the system uses additional checks to substitute for the limitations of traditional hash-based identification

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If storage systems increase deduplication checking to improve reliability, then fingerprint collision detection improves, but write latency increases

Engineering Contradiction:

Improvefingerprint collision detectionVSAvoidwrite latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent performs preliminary deduplication checks using hash functions before actual write operations to storage drives. By pre-processing and identifying duplicate data blocks before committing writes, the system reduces the need for extensive verification during the write process itself, thereby maintaining reliability while reducing write latency

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements a tiered verification approach where not all data blocks undergo the same level of checking. High-confidence matches from hash functions receive minimal verification, while ambiguous cases receive more thorough checking. This partial action approach maintains reliability for critical cases while reducing overall write latency

Inventive Principle:

Principle #16Partial or excessive action

3Reliability

If storage systems use more verification steps to handle fingerprint collisions, then data reliability improves, but device complexity increases

Engineering Contradiction:

Improvedata reliabilityVSAvoidverification process complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments the verification process into distinct stages: initial hash-based fingerprinting, collision detection phase, and verification phase. By dividing the complex verification process into manageable segments, the system achieves high data reliability while keeping each individual verification step relatively simple and efficient

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS10620864B1Improving the accuracy of in-line data deduplication

Publication Date: 2020.04.14 PURE STORAGE INC

AI summary

Handling fingerprint collisions in a storage system that includes one or more storage devices, including: determining whether a fingerprint associated with a first data segment matches a fingerprint associated with a data segment stored within the storage system; responsive to determining that the fingerprint associated with the first data segment matches the fingerprint associated with the data segment stored within the storage system, determining whether the first data segment matches the data segment stored within the storage system; and responsive to determining that the first data segment does not match the data segment stored within the storage system, resolving a fingerprint collision between the first data segment and the data segment stored within the storage system.