Storage Tiering via Segment Deduplication

Solution Overview

Problem

Conventional data systems face inefficiencies when moving files between storage tiers, as these operations often lose the benefits of deduplication and require significant resources, as they typically involve transferring entire files rather than just the necessary segments.

Innovation Solution

The method involves segmenting files into deduplicated units, allowing for the transfer of only missing segments between tiers while maintaining deduplication benefits, using segment indices and fingerprints for efficient identification and verification, and applying policies for tier movement based on file access and modification.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If files are moved between storage tiers by transferring entire files, then file movement between tiers is achieved, but resource usage increases and deduplication benefits are lost

Engineering Contradiction:

Improvefile movement efficiencyVSAvoidresource usage

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The patent segments files into smaller data blocks and uses deduplication to identify unique segments. When moving files between storage tiers, only the unique segments that don't already exist in the destination tier are transferred, rather than moving entire files. This segmentation approach maintains deduplication benefits and reduces resource usage during tier migration operations.

Inventive Principle:

Principle #1Segmentation

2Productivity

If files are moved between storage tiers by transferring entire files, then file movement between tiers is achieved, but deduplication benefits are lost

Engineering Contradiction:

Improvefile movement efficiencyVSAvoiddeduplication benefits

Core Design Contradiction:

ProductivityVSLoss of substance

Solution Approach 1:

The patent segments files into smaller data blocks and uses deduplication to identify unique segments. When moving files between storage tiers, only the unique segments that don't already exist in the destination tier are transferred, rather than moving entire files. This segmentation approach maintains deduplication benefits and reduces resource usage during tier migration operations.

Inventive Principle:

Principle #1Segmentation

3Productivity

If segment indices and fingerprints are used for efficient identification and verification, then transfer of only missing segments is achieved, but system complexity increases

Engineering Contradiction:

Improvesegment transfer efficiencyVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent creates simplified representations (fingerprints) of data segments that can be quickly compared to identify duplicates. Instead of comparing entire segments, the system uses these fingerprint copies to efficiently determine which segments need to be transferred, reducing the complexity of the comparison process while maintaining accuracy.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS10042855B2Efficient storage tiering

Publication Date: 2018.08.07 EMC IP HLDG CO LLC

AI summary

A method, article of manufacture, and apparatus for transferring data from a source tier to a target tier. In some embodiments, this may include dividing an object stored in the source tier into a plurality of segments, determining if the target tier is missing at least one of the segments, writing the missing segments to the target tier, and deleting the object from the source tier. In some embodiments, the object's metadata is updated to indicate that the object is in the target tier.