Continuous Data Protection De-duplication via Local Block Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Continuous data protection (CDP) solutions incur high costs due to inefficient data de-duplication processes, especially when backing up files to cloud-based storage, as they often require significant storage space and I/O operations, leading to increased fees for storage and transfer.
Innovation Solution
Implementing a CDP module that performs data de-duplication of new file versions against locally maintained previous versions before backing them up, using a collection of files with a master file containing unique data blocks and a version map file to minimize storage space and I/O operations on remote storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional file-level CDP solutions backup entire copies of modified files to remote storage, then data protection is ensured, but storage space consumption increases disproportionately
Solution Approach 1:
The patent segments files into fixed-size blocks and creates a block-level index structure. Instead of backing up entire files, only modified blocks are identified and backed up individually. This segmentation enables precise tracking of changes at the block level, allowing the system to backup only the necessary portions of files rather than complete copies.
Solution Approach 2:
The patent extracts only the modified blocks from complete files for backup purposes. By comparing block hashes between current and previous versions, the system identifies and extracts only the changed blocks, leaving unchanged blocks to be referenced from previous backups. This extraction approach dramatically reduces the volume of data sent to remote storage.
2Loss of substance
If de-duplication is performed against all previous versions stored on remote storage, then duplicate data is removed, but I/O requests and transfer costs increase
Solution Approach 1:
The patent performs preliminary de-duplication against the most recent previous version stored locally before initiating remote backup. By pre-identifying and removing duplicate blocks locally using the block index and hash comparison, the system minimizes the amount of data that needs to be transferred to remote storage. This preliminary action prevents unnecessary I/O operations and data transfer costs.
3Productivity
If entire previous version files are maintained locally for de-duplication, then de-duplication efficiency is improved, but local storage space requirements increase
Solution Approach 1:
Instead of storing entire previous version files locally, the patent stores only the block-level index structure and maintains blocks in a segmented manner. The index contains metadata about block locations and hashes, enabling efficient de-duplication checks without requiring complete file copies. This segmented approach preserves de-duplication efficiency while dramatically reducing local storage requirements.
Solution Approach 2:
The patent uses a simplified copy approach by maintaining only the essential block index structure locally rather than complete file copies. The index serves as a reference that enables de-duplication operations without requiring the actual file data to be present locally, thus achieving efficient de-duplication with minimal local storage overhead.
Data Source
AI summary
Technologies are described herein for performing data de-duplication of a version of a data file for backup to a remote storage location. A CDP module executing on a computer creates a collection of files corresponding to the version of the data file by de-duplicating the version against a previous version master file stored locally on the computer. The previous version master file contains one or more unique data blocks of a specific block size from a previous version of the data file. Once the de-duplication against the locally maintained previous version master file is complete, the CDP module stores the collection of files corresponding to the version of the data file to the remote storage location. The remote storage location also contains a master file corresponding to the data file that contains all of the unique data blocks in the previous version master file.