Centralized Index for Deduplicated Data Transfer

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional deduplication systems require re-assembly and re-deduplication of data chunks when transferring data between storage pools, making the process resource-intensive and inefficient.

Innovation Solution

Storing deduplication information at a storage management system level, allowing for the tracking and transfer of deduplicated data chunks across pools without the need for re-assembly or re-deduplication, by maintaining deduplication information in a centralized index accessible across all storage pools.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If deduplicated data is transferred between storage pools in conventional systems, then data transfer is achieved, but the process requires re-assembly and re-deduplication which increases resource consumption and processing time

Engineering Contradiction:

Improvedata transfer efficiencyVSAvoidresource consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary actions by maintaining deduplication information (chunk signatures, ownership, location) in a centralized database before transfer occurs. When data needs to be moved between pools, the system queries this pre-existing information to identify which chunks need to be transferred and which can be referenced remotely, eliminating the need for re-assembly and re-deduplication during the transfer process.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces a centralized database as an intermediary that stores deduplication information accessible by all storage pools. This intermediary contains chunk signatures, ownership information, and location data that mediates the transfer process between source and destination pools, allowing direct chunk transfer without re-processing at the destination.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Speed

If deduplicated data is transferred between storage pools, then data movement is achieved, but re-assembly and re-deduplication processes increase processing time

Engineering Contradiction:

Improvedata transfer speedVSAvoidprocessing time

Core Design Contradiction:

SpeedVSLoss of time

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent extracts the deduplication information (signatures, ownership, location data) from the storage pools and places it in a centralized database. This extraction allows the actual data transfer to bypass the time-consuming re-deduplication process, as the destination pool can directly use the pre-computed information to handle incoming chunks.

Inventive Principle:

Principle #2Taking out (Extraction)

3Reliability

If each storage pool maintains its own deduplication information independently, then local data integrity is ensured, but system-level data sharing and transfer efficiency are reduced

Engineering Contradiction:

Improvedata integrityVSAvoidsystem architecture complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent merges the distributed deduplication information from individual storage pools into a single centralized database. This consolidation maintains data integrity through centralized control while enabling efficient system-wide data sharing and transfer, as all pools access the same authoritative deduplication information rather than maintaining independent copies.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The centralized database serves multiple functions: it stores deduplication information for all pools, enables transfer operations between any pools, and provides a unified view of chunk ownership and location. This universal system replaces multiple independent deduplication systems while maintaining or improving data integrity across the entire storage infrastructure.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS8788466B2Efficient transfer of deduplicated data

Publication Date: 2014.07.22 WESTERN DIGITAL TECHNOLOGIES INC

AI summary

One aspect of the present invention includes enabling the efficient transfer of deduplicated data between storage pools in a storage management system without unnecessary reassembly and deduplication of data objects. In one embodiment, the storage management system tracks deduplication information for the data chunks of data objects within an index at the storage management system level, in addition to tracking storage information for each data object within another index at the storage management system level. The data chunk deduplication information is then accessible by any storage pool. Accordingly, transfers of the data objects and data chunks of the data object are easily facilitated, even between non-deduplicating and deduplicating storage pools.