Segmented De-duplication Index for Real-Time Storage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Data storage systems with large amounts of data for de-duplication face memory limitations, making it impossible to keep the entire index in memory, leading to slow real-time de-duplication processes.

Innovation Solution

Segmenting the de-duplication index into smaller segments, with only a subset loaded into memory at a time, and using predictive filters like Bloom filters to quickly determine if a block is indexed, allowing for real-time reference and efficient de-duplication.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If the entire de-duplication index is kept in memory, then real-time de-duplication can be achieved, but memory limitations prevent this for large data sets

Engineering Contradiction:

Improvede-duplication processing speedVSAvoidmemory capacity

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

The de-duplication index is divided into multiple segments that can be stored on disk rather than requiring the entire index to be in memory. Only the currently needed segment is loaded into memory for processing, enabling real-time de-duplication even with limited memory resources.

Inventive Principle:

Principle #1Segmentation

2Quantity of substance

If the index is stored on disk instead of in memory, then memory usage is reduced, but de-duplication becomes too slow for real-time processing

Engineering Contradiction:

Improvememory usageVSAvoidde-duplication processing speed

Core Design Contradiction:

Quantity of substanceVSSpeed

Solution Approach 1:

The needed index segment is pre-loaded into memory before the de-duplication operation begins. This preliminary action ensures that when de-duplication processing is needed, the index data is already in fast memory, avoiding slow disk access during the actual processing.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically manages memory by loading and unloading index segments as needed. The memory buffer is reused to hold different segments at different times, optimizing the balance between memory usage and processing speed.

Inventive Principle:

Principle #15Dynamics

3Productivity

If predictive filters are used to predict block indexing, then real-time de-duplication is enabled, but memory is required to store the filters

Engineering Contradiction:

Improvereal-time processing capabilityVSAvoidmemory for filter storage

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The predictive filter functionality is extracted from the main index structure and implemented as a separate, more compact component. This allows the filter to be stored in memory with minimal overhead while the main index remains on disk, enabling real-time processing without requiring large memory allocations.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS10614036B1Techniques for de-duplicating data storage systems using a segmented index

Publication Date: 2020.04.07 EMC IP HLDG CO LLC

AI summary

Techniques have been provided for storing data in a de-duplicated manner on a data storage system in a manner that allows for real-time reference to an index that is too large to fit within memory. This may be accomplished by segmenting the index into smaller segments, stored on disk. Only a subset of the segments may be loaded into memory at a given time. A predictive filter is stored in memory for each segment, allowing a de-duplication driver to quickly predict whether any given new block is likely to be indexed by each segment. Since identical blocks are often stored in long identical sequences (e.g., upon copying a disk image to a disk for a virtual machine), once a segment stored on disk is referenced many times in a short period, it is loaded into memory to allow the remainder of the long sequence to be de-duplicated.