Front-End Log Deduplication for Fast Write Acknowledgement

Solution Overview

Problem

Conventional deduplication approaches, such as inline and background deduplication, face inefficiencies and delays in data processing, particularly in acknowledging write operations and managing metadata for duplicate data storage.

Innovation Solution

Implementing a best-effort deduplication method that performs deduplication operations on data within a front-end log before it reaches back-end storage, allowing for immediate acknowledgement of write operations and reducing storage inefficiencies by generating digests and comparing them against existing data to identify duplicates, thereby eliminating redundant copies.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of substance

If conventional inline deduplication is performed during data processing along the I/O path, then duplicate data can be eliminated before storage, but write operation completion is delayed

Engineering Contradiction:

Improveduplicate data storageVSAvoidwrite operation completion time

Core Design Contradiction:

Loss of substanceVSLoss of time

Solution Approach 1:

The patent performs deduplication operations on data while it resides in the front-end log before the data is flushed to back-end storage. By generating digests and comparing them against existing data in the log, the system eliminates duplicate copies proactively. This preliminary action allows the system to acknowledge write operations immediately when data reaches the front-end log, without waiting for the slower back-end storage operations, thus resolving the contradiction between eliminating duplicates and maintaining fast write acknowledgment.

Inventive Principle:

Principle #10Preliminary action

2Loss of substance

If conventional background deduplication is performed by reading metadata from secondary storage, then duplicate data can be eliminated from storage, but processing efficiency is reduced due to metadata reading and evaluation

Engineering Contradiction:

Improveduplicate data in secondary storageVSAvoiddeduplication processing efficiency

Core Design Contradiction:

Loss of substanceVSProductivity

Solution Approach 1:

The patent performs deduplication operations on data while it resides in the front-end log before the data is flushed to back-end storage. By generating digests and comparing them against existing data in the log, the system eliminates duplicate copies proactively. This preliminary action allows the system to acknowledge write operations immediately when data reaches the front-end log, without waiting for the slower back-end storage operations, thus resolving the contradiction between eliminating duplicates and maintaining fast write acknowledgment.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent extracts the deduplication operation from the traditional background process that reads metadata from secondary storage and moves it to the front-end log processing stage. By taking out the deduplication function and applying it earlier in the data flow, the system avoids the inefficiency of reading and evaluating stored metadata from secondary storage, thereby improving processing efficiency while still eliminating duplicate data.

Inventive Principle:

Principle #2Taking out (Extraction)

3Reliability

If multiple copies of data are stored in back-end storage, then data availability is maintained, but storage space is wasted and subsequent deduplication processing is increased

Engineering Contradiction:

Improvedata availabilityVSAvoidstorage space consumption

Core Design Contradiction:

ReliabilityVSVolume of stationary object

Solution Approach 1:

The patent performs deduplication operations on data while it resides in the front-end log before the data is flushed to back-end storage. By generating digests and comparing them against existing data in the log, the system eliminates duplicate copies proactively. This preliminary action allows the system to acknowledge write operations immediately when data reaches the front-end log, without waiting for the slower back-end storage operations, thus resolving the contradiction between eliminating duplicates and maintaining fast write acknowledgment.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11194495B2Best-effort deduplication of data while the data resides in a front-end log along an I/O path that leads to back end storage

Publication Date: 2021.12.07 EMC IP HLDG CO LLC

AI summary

A technique performs best-effort deduplication. The technique involves activating a front-end log deduplication service that is configured and operative to perform deduplication operations on data in front-end log-based storage prior to that data reaching back-end storage that is different from the front-end log-based storage. The technique further involves, after the front-end log deduplication service is activated, receiving new data in the front-end log-based storage. The technique further involves, providing the front-end log deduplication service to perform a data deduplication operation on the new data while the new data resides within the front-end log-based storage. The technique further involves, after the data deduplication operation is performed on the new data, updating the back-end storage to indicate storage of the new data within the back-end storage.