Backup Storage System Fingerprint Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data backup between storage systems, there is a challenge in reducing the amount of duplicate data forwarded, as existing methods do not effectively identify and avoid forwarding duplicate data between backup-source and backup-destination storage systems, leading to inefficiencies in data transfer.
Innovation Solution
A system where the backup-source storage system extracts and registers patterns of forwarding target data, determining if they are already present in the backup-source storage system, and only forwards data that is not duplicate, using fingerprint values to identify and manage duplicate data chunks, thereby reducing the amount of data transferred.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is forwarded from backup-source storage system to backup-destination storage system without duplication check between systems, then all data is transferred ensuring complete backup, but the amount of forwarded data includes duplicates increasing transfer volume and time
Solution Approach 1:
The backup-source storage system performs preliminary duplication detection by comparing fingerprint values of data to be backed up with existing data in its storage units before forwarding data. This preliminary action identifies duplicates that already exist in the backup-source system, preventing unnecessary data transfer while ensuring that unique data is properly backed up to the backup-destination system.
Solution Approach 2:
The system uses fingerprint values as simplified copies or representations of actual data content for duplication detection. Instead of comparing entire data sets, the system compares compact fingerprint identifiers, enabling efficient duplicate detection without transferring full data volumes, thus reducing transfer volume while maintaining backup reliability.
2Productivity
If duplication detection is performed only within backup-source storage system, then duplicate data within the system is identified, but duplicate data between backup-source and backup-destination systems is not avoided
Solution Approach 1:
The backup-source storage system maintains a record of data fingerprints in its storage units and uses this feedback mechanism to detect duplicates. When data is prepared for backup, its fingerprint is compared against the recorded fingerprints, providing feedback on whether the data already exists in the system. This feedback enables the system to identify both internal duplicates and potential duplicates that would be recreated in the backup-destination system.
Solution Approach 2:
The fingerprint-based duplication detection mechanism serves multiple functions: it detects duplicates within the backup-source system, prevents duplicate data from being forwarded to the backup-destination system, and works across different storage units and data types. This universal approach handles various duplication scenarios (internal duplicates, restored data, reconfigured volume pairs) through a single mechanism.
3Reliability
If all restored data is copied to another data center for disaster recovery, then disaster recovery capability is ensured, but duplicate data is transferred unnecessarily increasing transfer time and volume
Solution Approach 1:
Before copying restored data to another data center for disaster recovery, the backup-source storage system performs preliminary duplication detection using fingerprint comparison. This preliminary action identifies which data already exists in the target data center, allowing the system to skip transferring duplicates while ensuring that unique restored data is properly copied, thus reducing transfer time while maintaining disaster recovery capability.
Data Source
AI summary
A backup-source storage system extracts, as a pattern, forwarding target data to a backup-destination storage system among data written to a storage apparatus through a volume. The backup-source storage system determines whether the extracted pattern matches with a registered pattern in the backup-source storage system. When a result of the determination is false, the backup-source storage system registers the extracted pattern and forwards the forwarding target data to the backup-destination storage system. When the result of the determination is true, the backup-source storage system does not forward the forwarding target data.