Deduplication System Checkpoint Error Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication systems face inefficiencies in large data file copying due to high metadata overhead and increased deduplication time, especially when dealing with large files, and they lack effective error handling mechanisms to ensure data integrity during copying, leading to potential data loss in disaster recovery scenarios.
Innovation Solution
The system employs a fingerprint-based approach where 20-byte SHA-1 hash fingerprints are generated for data segments, allowing for efficient comparison and deduplication, and implements a generation identifier mechanism to track service instances and restart from checkpoints, ensuring data integrity and reducing the need for full file retransmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If data files are divided into small data segments for deduplication, then more duplicates can be identified and storage space is saved, but metadata size increases and deduplication time increases
Solution Approach 1:
The patent divides large data files into smaller data segments for processing. This segmentation enables the deduplication system to identify duplicates more effectively at the segment level while managing metadata overhead through selective processing of unique segments only.
Solution Approach 2:
The patent extracts and processes only the metadata necessary for deduplication identification, separating essential deduplication information from complete segment data. This extraction approach reduces metadata processing time while maintaining duplicate detection accuracy.
2Loss of substance
If data files are divided into small data segments for deduplication, then more duplicates can be identified and storage space is saved, but the total size of metadata increases
Solution Approach 1:
The patent extracts only the essential metadata required for deduplication identification, separating necessary deduplication information from complete segment data. This extraction approach reduces metadata processing time while maintaining duplicate detection accuracy.
Solution Approach 2:
Instead of storing complete metadata for all segments, the patent inverts the approach by storing only references to unique segments and their fingerprints, significantly reducing metadata size while preserving deduplication functionality.
3Reliability
If the copying process restarts from the beginning after an error is detected, then data integrity can be verified, but copying time increases significantly for large files
Solution Approach 1:
The patent performs preliminary actions by generating and storing fingerprints for data segments during the initial copying process. These fingerprints serve as verification markers that enable error detection without requiring complete re-copying, thus maintaining data integrity while reducing recovery time.
Solution Approach 2:
The patent uses fingerprint copies of data segments as verification markers. Instead of copying and verifying entire data segments, the system copies and verifies compact fingerprint representations, enabling efficient error detection and recovery for large files.
4Reliability
If all fingerprints and segments are retransmitted after an error, then data integrity can be ensured, but the time required increases for large data files
Solution Approach 1:
The patent extracts and retransmits only the specific data segments and their fingerprints that contain errors, rather than retransmitting all fingerprints and segments. This selective extraction approach maintains data integrity while significantly reducing retransmission time for large files.
Solution Approach 2:
The patent uses fingerprint copies to identify and retrieve only the necessary data segments for retransmission. By copying and comparing fingerprints, the system determines exactly which segments need retransmission, avoiding unnecessary data transfer.
Data Source
AI summary
Object service receives communication of fingerprints stream, corresponding to file segments, from file source, and identifies sequential fingerprints in fingerprints stream as fingerprints group. Object service identifies group identifier for fingerprints group, and communicates fingerprints group to deduplication service associated with group identifier range including group identifier. Deduplication service identifies fingerprints in fingerprints group which are missing from fingerprint storage, and communicates identified fingerprints to object service, which communicates request for file segments, corresponding to identified fingerprints, to file source. Deduplication service receives communication of requested segments from file source, and stores requested segments. System identifies generation identifier associated with time of communicating by object service or deduplication service and identifies generation identifier associated with another time of communicating by object service or deduplication service. If generation identifier associated with time differs from generation identifier associated with other time, object service or deduplication service restarts communication.