Multi-Client Backup Deduplication via File-Level Chunking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing backup systems face inefficiencies in storage space utilization due to high metadata-to-data overhead ratios and increased hash collisions when deduplicating block-level backups across multiple client file systems, leading to performance issues.
Innovation Solution
The Multi-Client Backup Deduplication Apparatus employs a combination of content-aware chunking and fixed offset chunking to create manifest files for NTFS file systems, allowing for efficient deduplication by storing a single copy of identical data blocks and optimizing storage and retrieval processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If block-level deduplication is used across multiple client file systems, then storage space utilization is improved, but metadata-to-data overhead ratio increases and hash collisions occur
Solution Approach 1:
The patent segments data at the file level rather than block level, creating file-level deduplication. Each file is processed as a complete unit, and only the first occurrence of identical files is stored, while subsequent identical files are replaced with references to the original. This segmentation approach reduces metadata overhead compared to block-level deduplication because it handles complete files as atomic units rather than breaking them into smaller blocks that require extensive metadata tracking.
Solution Approach 2:
The patent uses reference copying instead of actual data copying for duplicate files. When a duplicate file is encountered, instead of copying the entire file data again, the system creates a reference pointer to the original file's location in storage. This copying approach significantly reduces storage space requirements while minimizing metadata overhead since only the reference information needs to be stored, not the complete duplicate data.
2Quantity of substance
If block-level deduplication is used, then storage space is reduced, but performance deteriorates due to increased hash collisions
Solution Approach 1:
By segmenting at the file level rather than block level, the patent reduces the number of items that need to be hashed and compared. File-level deduplication processes complete files as single units, which reduces the total number of hash computations required compared to block-level processing. This segmentation approach directly improves performance by reducing hash collision probability and decreasing processing overhead.
Solution Approach 2:
The patent changes the deduplication parameter from block size to file size. Instead of using a fixed block size for deduplication, the system uses complete files as the unit of deduplication. This parameter change fundamentally alters how deduplication works, reducing the number of hash operations and improving performance while maintaining effective storage space utilization.
3Quantity of substance
If content-aware chunking is used for deduplication, then storage efficiency is improved, but system complexity increases
Solution Approach 1:
The patent implements content-aware chunking by segmenting files into meaningful content units rather than fixed blocks. The system analyzes file content to identify logical segments that can be independently deduplicated, which improves storage efficiency by capturing semantic similarities between files. This segmentation approach handles complex scenarios like partial file duplicates and versioning more effectively.
Solution Approach 2:
The patent changes the chunking parameter from fixed size to content-aware dynamic sizing. Instead of using a predetermined block size, the system dynamically determines chunk boundaries based on content analysis, file structure, and semantic similarity. This parameter change enables more intelligent deduplication that adapts to different file types and content patterns, improving storage efficiency while managing complexity through adaptive rather than rigid chunking.
Data Source
AI summary
The Multi-Client Backup Deduplication Apparatuses, Methods and Systems (“MCBD”) transforms backup request, restore request, virtual machine initialization request inputs via MCBD components into backup response, restore response, virtual machine initialization response outputs. A backup request identifying a source volume is obtained. The source volume's master file table is retrieved and a file entry is selected. File data runs associated with the selected entry are determined. The file is reassembled in a buffer and split into file chunks Upon determining that a file chunk is not indexed in a chunk index, the file chunk is stored on a target volume and indexed in the chunk index. File chunk slice datastructures that map source volume offset location of file data on the source volume to the corresponding file chunk offset location of that file data in the file chunk on the target volume are generated and stored in a manifest file.