File Copying Checkpoint and Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face inefficiencies in large data file copying due to high metadata overhead, increased deduplication time, and data fragmentation, especially when dealing with large files, and struggle with error handling in file copying processes, leading to potential data loss during disaster recovery scenarios.
Innovation Solution
The system employs a method that generates fingerprints for data segments, uses similarity group identifiers to route L1 segments efficiently, and implements checkpointing to resume file copying from the last valid state upon errors, reducing redundant data transfer and minimizing data loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data files are divided into small data segments for deduplication, then more duplicates can be identified and storage space is saved, but metadata overhead increases, deduplication time increases, and data fragmentation increases
Solution Approach 1:
The patent divides large data files into smaller data segments that can be processed independently and in parallel. This segmentation enables the deduplication system to identify duplicates more effectively across multiple segments while allowing parallel processing to reduce overall deduplication time. The segmented approach also reduces metadata overhead by processing manageable portions of data separately.
Solution Approach 2:
The patent implements preprocessing steps including data normalization and segment organization before the actual deduplication process. By preparing data in advance with consistent formatting and structure, the system reduces the computational complexity during deduplication, thereby decreasing processing time while maintaining effective duplicate identification across segmented data.
2Quantity of substance
If data files are divided into small data segments for deduplication, then more duplicates can be identified and storage space is saved, but metadata overhead increases
Solution Approach 1:
The patent merges multiple small data segments back together after deduplication processing to form reconstructed data files. This merging approach allows the system to maintain the benefits of segment-based deduplication (reduced storage requirements through duplicate identification) while minimizing metadata overhead by consolidating processed segments into unified output structures.
Solution Approach 2:
The patent uses reference pointers instead of actual data copies to represent duplicate segments. Rather than storing redundant metadata for each duplicate segment, the system creates lightweight reference copies that point to the original unique segment, significantly reducing metadata overhead while maintaining the ability to reconstruct complete files.
3Reliability
If data file copying restarts from the beginning upon error detection, then data integrity is verified, but copying time increases significantly for large files
Solution Approach 1:
The patent implements checkpoint creation at predetermined intervals during the file copying process. These checkpoints serve as pre-prepared recovery points that allow the system to verify data integrity through checksum validation without requiring a complete restart. If errors are detected, the copying process can resume from the last valid checkpoint, maintaining reliability while minimizing time loss.
Solution Approach 2:
The patent enables continuous copying operation by implementing error handling that resumes processing from checkpoint rather than restarting. This continuity allows the system to maintain data integrity through verification while avoiding the significant time penalty of complete restarts, especially for large files where checkpoints are strategically positioned to minimize reprocessing.
4Loss of time
If checkpoints are created during file copying, then error recovery time is reduced, but system complexity and storage overhead increase
Solution Approach 1:
The patent extracts checkpoint information into separate metadata structures that are stored independently from the main data copying logic. This extraction simplifies the core copying process while allowing checkpoint management to handle the complexity of error recovery. The separated checkpoint data structures reduce system complexity by isolating recovery mechanisms from the primary data transfer operations.
Data Source
AI summary
Object service receives request to copy file to destination and identifies group identifier for fingerprints group corresponding to sequential segments in file. Object service communicates request for fingerprints group to deduplication service associated with group identifier range including group identifier. Deduplication service communicates fingerprints group, retrieved from fingerprint storage, to object service, which communicates fingerprints group and group identifier to destination. Object service communicates request for file segments, corresponding to fingerprints missing in destination, communicated from destination, to deduplication service, which communicates requested segments, retrieved from source storage, to object service, which communicates requested segments to destination. System identifies generation identifier associated with time of communicating by object service or deduplication service, and generation identifier associated with another time of communicating by object service or deduplication service. If generation identifier associated with time differs from generation identifier associated with other time, object service or deduplication service restarts communication.