Distributed De-duplication Storage Controller Fragmentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data storage systems face challenges in efficiently managing storage in distributed de-duplication systems due to bandwidth and storage capacity constraints, requiring complex architectures and scalable shared databases that are difficult to develop and maintain, especially when scaling system capacity and geographic scope.

Innovation Solution

A method for managing storage in a distributed de-duplication system involving storage controllers that cache and fragment data, compute hash values, and use tenant keys to identify and encrypt fragments, thereby avoiding storage of duplicates and optimizing bandwidth and storage usage without requiring modifications to the storage repositories.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If prior de-duplication architectures are implemented, then storage utilization is improved, but system complexity increases due to deep integration requirements, low-latency distributed databases, and complex fault recovery systems

Engineering Contradiction:

Improvestorage utilizationVSAvoidsystem complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent extracts the de-duplication logic from the storage system core and relocates it to the data source. By computing fingerprints at the source and storing only unique data blocks, the system eliminates the need for complex centralized de-duplication databases and fault recovery systems while maintaining storage efficiency.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent introduces an intermediary mechanism where data sources themselves perform fingerprint computation and duplicate detection. This intermediary approach at the source end eliminates the need for complex centralized coordination, reducing system complexity while achieving the same de-duplication goals.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If scalable shared databases storing fingerprints are created, then data identification accuracy is improved, but development and maintenance difficulty increases especially when scaling system capacity and geographic scope

Engineering Contradiction:

Improvedata identification accuracyVSAvoiddevelopment and maintenance difficulty

Core Design Contradiction:

Measurement precisionVSEase of manufacture

Solution Approach 1:

The patent enables data sources to self-serve by computing their own fingerprints and performing duplicate detection locally. This eliminates the need for centralized fingerprint databases, allowing systems to scale geographically without increasing database development and maintenance complexity while maintaining accurate data identification.

Inventive Principle:

Principle #25Self-service

3Quantity of substance

If large amounts of data are exchanged between client and server, then data completeness is improved, but bandwidth consumption and usage costs increase

Engineering Contradiction:

Improvedata completenessVSAvoidbandwidth consumption

Core Design Contradiction:

Quantity of substanceVSLoss of energy

Solution Approach 1:

The patent performs preliminary fingerprint computation and duplicate detection at the data source before data transmission. By identifying and eliminating duplicates in advance, the system ensures data completeness is maintained while significantly reducing the volume of data that needs to be transmitted over the network.

Inventive Principle:

Principle #10Preliminary action

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

This approach minimizes CPU and memory usage, avoids storing duplicate data, and supports hybrid cloud de-duplication with multi-tenant security, improving namespace migration and storage efficiency while maintaining data security and reducing operational complexity.

Implementation Method 1

a storage controller receives, from a data source, data to be stored into a storage repository

Methodology Applied

Scientific EffectHash computation:

Implementation Method 2

use tenant keys to identify and encrypt fragments, thereby avoiding storage of duplicates

Methodology Applied

Scientific EffectEncryption:

Data Source

PatentUS11797488B2Methods for managing storage in a distributed de-duplication system and devices thereof

Publication Date: 2023.10.24 NETAPP INC

AI summary

A method, non-transitory computer readable medium, and device that assists with managing storage in a distributed deduplication system includes receiving an object to be stored from a client computing device. The received object is divided into a plurality of fragments. A plaintext hash value and a ciphertext hash value is determined for each of the plurality of fragments, wherein each of the plurality of fragments is renamed with the corresponding determined ciphertext hash value. Each of the renamed plurality of fragments are stored in a plurality of storage repositories.