Local Dedupe Restore for Corrupted Remote-Backed Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup systems waste computing resources by fetching corrupted files from remote backups when local duplicates exist, especially in environments with high file-level deduplication, and fail to efficiently restore files with matching metadata.
Innovation Solution
Implement a local dedupe engine that maintains a file catalog with local and remote paths and hashes, allowing local file restoration using local duplicates and remote metadata retrieval, conserving resources by reducing remote data transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system fetches corrupted files from remote backups, then file restoration is achieved, but computing resources and network bandwidth are wasted when local duplicates are available
Solution Approach 1:
The system performs preliminary actions by maintaining a file catalog with hash values and identifying local duplicate files before restoration is needed. When corruption is detected, the system can immediately utilize pre-identified local duplicates without needing to fetch files from remote backups, thus preventing waste of computing resources and network bandwidth.
Solution Approach 2:
The system enables self-service by using locally stored duplicate files to repair corrupted files without requiring external intervention from remote backup systems. The file catalog and hash matching mechanism allow the system to autonomously identify and utilize local resources for restoration, eliminating the need to transfer data over the network when local duplicates exist.
2Reliability
If the system transfers data from remote clusters, then file restoration is achieved, but network bandwidth is consumed
Solution Approach 1:
The system extracts only the necessary information (hash values and file metadata) to store in the file catalog, rather than transferring entire files. When restoration is needed, the system compares hash values to identify local duplicates, extracting only the minimal necessary data from remote backups when local copies are unavailable, thus conserving network bandwidth.
Solution Approach 2:
The system creates and maintains copies of file hash values and metadata in the local file catalog, enabling rapid identification of duplicate files without needing to transfer actual file data. This copying approach allows the system to restore files using local duplicates when available, avoiding network transfers entirely.
3Device complexity
If the system does not use local duplicates for repair, then remote backup systems are simplified, but computing resources are wasted
Solution Approach 1:
The system performs preliminary actions by pre-computing and storing file hash values in the file catalog before restoration is needed. This allows the system to efficiently identify local duplicates using simple hash comparisons rather than complex file analysis, improving computing resource efficiency without requiring changes to the remote backup system architecture.
Solution Approach 2:
The system changes the parameter of file identification from comparing entire file contents to comparing hash values. This parameter change enables efficient duplicate detection using minimal computational resources, allowing the system to utilize local duplicates effectively without increasing remote backup system complexity.
Data Source
AI summary
A system can maintain a backup of data stored on a source computer, wherein the backup is stored on a remote computer. The system can maintain, on the source computer, a file catalog, wherein the file catalog comprises a local path for a first file on the source computer, a remote backup location for the first file on the remote computer, and a first hash of the first file. The system can determine that a copy of the first file at the local path on the source computer is corrupted. The system can identify whether a second hash of a second file on the source computer matches the first hash of the first file. The system can, in response to determining that the second hash matches the first hash, repair first data of the first file with second data of the second file, and repair first metadata of the first file with second metadata from the remote backup location for the first file on the remote computer.