Storage Replication Using Data Fingerprints for Database Failure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data replication processes become inefficient when the mapping database between source and destination storage systems becomes unavailable or inaccurate, leading to unnecessary data transfers due to the inability to detect already received data.
Innovation Solution
The destination storage system uses data fingerprints to identify similar data blocks and verifies their identity without transferring data, updating the mapping database to prevent duplication and maintain replication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a mapping database is used to track data between storage systems, then data replication efficiency is improved, but the system becomes vulnerable to database unavailability or inaccuracy
Solution Approach 1:
The patent creates a fingerprint (hash copy) of the data that can independently verify data identity without requiring the original mapping database. This fingerprint copy enables the system to detect duplicate data even when the mapping database is unavailable or inaccurate, resolving the contradiction between replication efficiency and database reliability.
Solution Approach 2:
The fingerprint acts as an intermediary between the data and the mapping database, providing an alternative verification mechanism. When the mapping database fails, the fingerprint intermediary enables continued operation by independently verifying data identity, thus maintaining system reliability while preserving replication efficiency.
2Productivity
If data is transferred without verification when mapping database is unavailable, then replication continues, but redundant data transfers occur
Solution Approach 1:
The patent replaces the mechanical dependency on the mapping database with a computational fingerprint verification system. Instead of relying on database lookups to prevent redundant transfers, the system uses fingerprint comparison to detect duplicates, enabling replication continuity while avoiding redundant data transfers even when the database is unavailable.
Solution Approach 2:
The system performs preliminary fingerprint verification before data transfer to determine if data already exists at the destination. This preliminary action prevents redundant transfers from occurring in the first place, maintaining replication continuity while avoiding the energy waste of unnecessary data transfers.
3Measurement precision
If fingerprint verification is performed for all data, then accuracy of duplicate detection is improved, but processing time increases
Solution Approach 1:
The patent applies fingerprint verification selectively rather than universally - using it primarily when the mapping database is unavailable or inaccurate. This partial application maintains high duplicate detection accuracy when needed while avoiding unnecessary processing time overhead when the database is functioning normally.
Data Source
AI summary
A system and method for data replication is described. A destination storage system receives a message from a source storage system as part of a replication process. The message includes an identity of a first file, information about where the first file is stored in the source storage system, a name of a first data being used by the first file and stored at a first location of the source storage system, and a fingerprint of the first data. The destination storage system determines that a mapping database is unavailable or inaccurate, and accesses a fingerprint database using the fingerprint of the first data received with the message to determine whether data stored in the destination storage system has a fingerprint identical to the fingerprint of the first data.