Variable-Length Chunking for Backup Deduplication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional deduplicated data systems face inefficiencies in storing and retrieving data due to fixed-width chunking, which results in large amounts of identical information not being deduplicated and high metadata overhead, and differential compression methods that require significant decompression demands and metadata storage, limiting direct data access and retrieval.

Innovation Solution

A computer-implemented data backup method that stores metadata in a secondary storage layer and divides data into variable-length chunks, checking for existing chunks in a primary storage layer to provide an indication of their location, enabling efficient inline deduplication and differential compression, and allowing direct access and restoration without proprietary formats.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If fixed-width chunking is used for deduplication, then storage simplicity is improved, but deduplication effectiveness deteriorates because identical sequences at arbitrary offsets cannot be recognized

Engineering Contradiction:

Improvechunking structureVSAvoiddeduplication effectiveness

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The patent segments data into variable-width chunks instead of fixed-width chunks, allowing the system to identify identical data sequences regardless of their position within the file. This segmentation approach enables the deduplication algorithm to compare chunks of varying sizes, thereby capturing identical sequences that span different offsets in source and destination files.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the parameter of chunk width from fixed to variable, allowing chunks to expand or contract based on the actual data patterns being compared. This parameter change enables the system to adaptively match identical sequences regardless of their position in the file, resolving the contradiction between storage simplicity and deduplication effectiveness.

Inventive Principle:

Principle #35Parameter changes

2Reliability

If variable-width chunks are used to improve deduplication effectiveness, then deduplication capability is improved, but metadata overhead increases

Engineering Contradiction:

Improvededuplication effectivenessVSAvoidmetadata overhead

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent uses hashing to create compact representations (copies) of variable-width chunks for comparison purposes. Instead of storing and comparing entire variable-width chunks which would generate excessive metadata, the system computes hash values that serve as compact identifiers, enabling efficient deduplication with minimal metadata overhead.

Inventive Principle:

Principle #26Copying

3Loss of substance

If differential compression is applied to multiple similar blocks, then compression ratio is improved, but decompression complexity and metadata requirements increase

Engineering Contradiction:

Improvedata redundancyVSAvoiddecompression process

Core Design Contradiction:

Loss of substanceVSDevice complexity

Solution Approach 1:

The patent performs deduplication and compression operations during the data writing phase rather than during retrieval. By identifying identical chunks and storing only unique copies upfront, the system eliminates the need for complex decompression operations later, as the data is already in its most compact form.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent extracts only the unique portions of data and stores references to identical chunks rather than storing multiple copies. This extraction approach reduces the amount of data that needs to be processed during compression and retrieval, simplifying the overall system complexity while maintaining high compression ratios.

Inventive Principle:

Principle #2Taking out (Extraction)

4Productivity

If data is stored in proprietary deduplication format, then deduplication efficiency is improved, but direct data access without backup application is prevented

Engineering Contradiction:

Improvededuplication efficiencyVSAvoiddata accessibility

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent implements a dual-format storage system where data is maintained both in its original format for direct access and in a deduplicated format for efficient storage. This multi-functionality allows the system to serve both purposes: achieving high deduplication efficiency while simultaneously enabling direct data access without requiring the backup application, as the original data format remains accessible.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12141036B2Data backup system and method in fixed size deduplication systems

Publication Date: 2024.11.12 HUAWEI TECH CO LTD

AI summary

A computer-implemented data backup method comprises storing metadata associated with an object in a secondary storage layer of a backup system, the metadata representing a data structure for the object, dividing a set of data representing the object into multiple variable length chunks, determining whether an existing chunk of data that is the same as at least a part of the variable length chunk exists in a primary storage layer of the backup system, storing the variable length chunk in the primary storage layer of the backup system, and on the basis of the determination, providing the primary storage layer of the backup system with an indication of the location of the existing chunk of data.