Backup Server Watermark Versioning for Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup systems face inefficiencies in storage due to redundant data duplication and resource-intensive hashing algorithms, which lead to increased storage consumption and processing power requirements, and may result in corrupted backup data sets due to hash collisions.
Innovation Solution
Implementing a system where a backup server maintains multiple versions of data blocks and uses watermarks to identify changes, allowing for incremental backups without hashing large data sets, and performs de-duplication across clients and data versions to reduce storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full backup sets are stored at each backup interval, then data recovery reliability is improved, but storage space consumption increases significantly due to duplicating unchanged data
Solution Approach 1:
The backup data is segmented into fixed-size chunks, and each chunk is independently hashed and tracked. This allows the system to identify and store only the changed chunks rather than entire files or backup sets, resolving the contradiction between ensuring complete data recovery and minimizing storage consumption of unchanged data.
Solution Approach 2:
Instead of copying entire backup sets, the system creates selective copies of only the changed data chunks identified through hash comparison. This maintains the ability to reconstruct full backup sets when needed while dramatically reducing the storage space required for incremental backups.
2Quantity of substance
If hashing algorithms are used to identify changed data, then storage efficiency is improved by reducing redundant backups, but processing power and I/O resources increase significantly
Solution Approach 1:
The system generates hash values for data chunks during the initial full backup operation, storing these hash values in an index. This preliminary action eliminates the need to re-hash data during incremental backups, reducing processing power and I/O resource requirements while maintaining storage efficiency through hash-based change detection.
3Quantity of substance
If hash-based approaches are used to identify incremental data, then storage consumption is reduced, but data corruption risk increases due to hash collisions
Solution Approach 1:
The system changes the parameter used for change detection from simple hashing to a composite identification method that includes hash values plus additional metadata (such as file identifiers, chunk positions, and version information). This multi-parameter approach maintains the storage consumption benefits of hashing while eliminating data corruption risks by making hash collisions extremely unlikely and providing multiple verification points.
4Adaptability or versatility
If multiple full backup sets are maintained, then data recovery options are improved, but the complexity of managing and storing backup data increases
Solution Approach 1:
The backup system segments data into chunks and maintains an index that tracks the location and version of each chunk across multiple backup sets. This segmentation approach allows the system to provide multiple data recovery options (restoring any previous version of any file) while simplifying management through the centralized index, rather than managing complete backup set structures.
Data Source
AI summary
According to embodiments described herein, a backup server maintains backup data for a set of data, which includes data for a first block and a second block. Backup data for the first and second block include backup data for a plurality of versions of the first and second block. A distinct watermark is stored for each version of the first block and each version of the second block. In response to a request to perform a restoration operation on the set of data, a particular version of the first block and a particular version of the second block are selected to use in the restoration operation by comparing a restoration target with the watermarks of the version of the first block and second block. The selected version of the first block has a different watermark than the selected version of the second block.