Deduplication Hit Rate Optimization in Remote Copy Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in deduplication and data management across replicated systems, particularly when resources like memory are limited, leading to suboptimal deduplication hit rates and increased storage costs.
Innovation Solution
The method involves calculating cryptographic values for data and using fingerprint and forward lookup structures to identify and update entries in both primary and backup systems, allowing for pre-population of deduplication databases and optimizing deduplication hit rates by directing the remote system to look for potential deduplication data, thereby reducing the dependency on the remote system's capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the remote system performs independent deduplication with limited memory resources, then it can operate autonomously, but the deduplication hit rate decreases due to insufficient lookup capacity
Solution Approach 1:
The patent combines the primary system's deduplication findings with the backup system's autonomous deduplication by transmitting cryptographic values and address mappings from the primary system to the backup system. This merging allows the backup system to leverage the primary system's larger memory resources for deduplication lookups while maintaining autonomous operation, thereby improving the deduplication hit rate without sacrificing operational independence.
Solution Approach 2:
The primary system performs deduplication operations and stores findings (cryptographic values and address mappings) in advance before the backup system needs them. By pre-calculating and transmitting this deduplication information, the backup system can quickly perform lookups using received address mappings without needing to maintain large memory resources itself, thus improving deduplication effectiveness while operating with limited resources.
2Loss of energy
If the primary system transmits only cryptographic values to the backup system, then network bandwidth is conserved, but the backup system cannot achieve optimal deduplication without scanning its entire directory
Solution Approach 1:
The primary system performs the time-consuming directory scanning and deduplication analysis in advance, then transmits only the essential results (cryptographic values and address mappings) to the backup system. This preliminary action allows the backup system to quickly perform deduplication lookups using the pre-computed address mappings without needing to scan its entire directory, thus improving deduplication processing speed while keeping network bandwidth consumption low.
Solution Approach 2:
The patent extracts only the critical deduplication information (cryptographic values and address mappings) from the primary system's complete directory data and transmits only this extracted subset to the backup system. This extraction approach enables the backup system to achieve effective deduplication without receiving or processing the entire directory structure, thereby improving processing speed while minimizing network bandwidth usage.
3Measurement precision
If the backup system maintains a complete local copy of the directory for independent deduplication, then it achieves high deduplication hit rate, but memory resource consumption increases
Solution Approach 1:
The primary system pre-computes and transmits address mappings that enable the backup system to perform effective deduplication lookups without maintaining a complete local directory copy. By receiving these pre-computed mappings, the backup system can achieve high deduplication hit rates using minimal memory resources, as it only needs to store the transmitted mappings rather than scanning or maintaining its entire directory structure.
Solution Approach 2:
Instead of copying the entire directory structure to the backup system, the patent transmits only the essential deduplication information (cryptographic values and address mappings). This selective copying approach allows the backup system to achieve effective deduplication with a minimal data footprint, thereby improving deduplication hit rate while keeping memory resource consumption low.
4Measurement precision
If the primary system transmits address mappings to the backup system, then the backup system achieves high deduplication hit rate, but network bandwidth consumption increases
Solution Approach 1:
The patent extracts and transmits only the essential deduplication information (cryptographic values and address mappings) from the primary system to the backup system, excluding redundant directory structure data. This extraction approach enables the backup system to achieve high deduplication hit rates while minimizing the amount of data transmitted over the network, thereby reducing network bandwidth consumption.
Solution Approach 2:
The primary system performs deduplication analysis in advance and transmits the results (address mappings) to the backup system before actual data replication occurs. This preliminary transmission of essential information allows the backup system to achieve high deduplication hit rates without requiring continuous or repeated network transmissions, thus reducing overall network bandwidth consumption.
Data Source
AI summary
A method, system, computer program product and computer program for managing a storage system, comprising a primary system and a backup system, wherein the backup system is in a copy relationship with the primary system, comprising: in response to a write operation, the write operation comprising first data: calculating a cryptographic value for the first data; scanning a first directory to identify an entry corresponding to the cryptographic value to determine a first set of addresses associated with the write operation; transmitting the first set of addresses to the backup system; and updating the first directory with a first entry for the write operation, the first entry comprising a pointer to the first set of addresses; updating a second directory with a second entry for the write operation, the second entry comprising a pointer to a second set of addresses corresponding to the first set of addresses.