Container-Based Data Deduplication for Backup Restoration Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing backup systems face slow restoration times due to chunk fragmentation in deduplicated backups, where new chunks are stored sequentially, leading to scattered data locations and increased disk seek operations, resulting in sluggish retrieval speeds.
Innovation Solution
Implementing a container-based deduplication system that uses chunk containers as on-disk data structures, allowing for efficient storage and retrieval of data chunks by grouping similar chunks together, reducing the need for frequent disk seeks through a locality-based assignment algorithm.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If new chunks are stored sequentially in the backup system, then storage organization is simplified, but chunks from the same backup become scattered across storage leading to slow retrieval
Solution Approach 1:
The patent segments the backup storage into multiple containers, each dedicated to storing chunks from a specific backup. This segmentation allows chunks from the same backup to be grouped together in the same container, improving retrieval speed by reducing disk seek operations while maintaining simplified storage organization through the structured container approach.
Solution Approach 2:
The patent introduces a new organizational dimension by creating containers that group chunks not just by sequential storage location but by backup source. This additional dimensional organization (containers as intermediaries between storage and retrieval) resolves the contradiction by enabling both simplified storage management and fast retrieval through localized chunk access.
2Quantity of substance
If deduplication is implemented to reduce storage requirements, then storage efficiency improves, but chunk fragmentation increases causing slower restoration
Solution Approach 1:
The patent segments deduplicated chunks into containers organized by backup source, allowing efficient storage utilization through deduplication while maintaining restoration speed by keeping related chunks together. Each container acts as a localized storage unit that preserves chunk coherence even as deduplication distributes chunks across the storage system.
Solution Approach 2:
The container serves as an intermediary structure between the deduplication storage mechanism and the restoration process. It mediates between the need for storage efficiency (through deduplication) and restoration speed (through localized access), allowing chunks to be stored efficiently while being retrieved in an organized manner.
3Quantity of substance
If chunks are distributed across different locations for deduplication, then storage space is optimized, but disk seek operations increase reducing retrieval performance
Solution Approach 1:
The patent segments the storage system into containers that locally hold chunks from specific backups, reducing the need for disk seek operations during retrieval. This segmentation allows storage space efficiency through deduplication while minimizing disk seek time by keeping chunks from the same backup together in the same container.
Solution Approach 2:
The patent applies local quality by organizing chunks within containers based on their backup source, creating localized regions of related data. This local organization reduces disk seek operations for retrieval while maintaining overall storage space efficiency through deduplication across the entire system.
Data Source
AI summary
A method for data deduplication includes receiving a set of hashes derived from a data chunk of a set of input data chunks 310. The method includes sampling the set of hashes 320, using an index indentifying data chunk containers that hold data chunks having a hash in the set of sampled hashes 330, and loading indexes for at least one of the identified data chunk containers 340. The method includes determining which of the hashes correspond to data chunks stored in data chunk containers corresponding to the loaded indexes 350 and deciding which of the set of input data chunks should be stored based at least in part on the determination.