Cooperating Deduplicators for Data Redundancy Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication in replica systems requires repetitive processing and increased computational resources due to the need to replicate deduplication work from multiple primary systems, leading to inefficiencies in data storage and management.
Innovation Solution
Implementing a system with cooperating deduplicators, where a primary deduplicator and a secondary deduplicator communicate to determine duplicate data segments, allowing the secondary deduplicator to avoid redundant processing and efficiently store segments, using segment IDs and reference counters to manage data storage and deletion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If a replica system replicates data from multiple primary systems with individual deduplication, then data redundancy is reduced for each primary system, but the replica system must repeat deduplication processing for all primary systems, increasing computational overhead
Solution Approach 1:
The patent merges the deduplication operations of multiple primary systems into a single coordinated deduplication process at the replica system. Instead of each primary system performing independent deduplication and the replica system repeating the work, the invention combines their deduplication efforts by having the replica system receive data segments from multiple primaries and perform a unified deduplication check against a shared segment database, eliminating redundant processing while maintaining the redundancy reduction benefits for all systems
Solution Approach 2:
The replica system's deduplication mechanism is designed to serve multiple primary systems simultaneously through a universal segment database that stores segments from all primaries. This multi-functional approach allows the same deduplication infrastructure to handle data from multiple sources, reducing the need for separate deduplication processes for each primary system and thereby reducing overall computational overhead while maintaining data redundancy reduction
2Quantity of substance
If deduplication is implemented in a replica system serving multiple primary systems, then storage efficiency is improved, but processing time increases due to repeated deduplication work
Solution Approach 1:
The patent implements preliminary action by having the replica system maintain a pre-populated segment database containing segments from all primary systems before new data arrives. When new data segments are received, the system performs quick lookup checks against this pre-established database rather than performing full deduplication analysis, significantly reducing processing time while maintaining storage efficiency
Solution Approach 2:
The system uses copying by creating segment identifiers (fingerprints) that represent data segments without storing the actual segment data multiple times. When the same segment is received from different primary systems, the system copies the segment identifier rather than the full segment data, enabling fast comparison and deduplication decisions that reduce processing time while maintaining storage efficiency
Data Source
AI summary
A method of determining whether a data segment is a duplicate using cooperating deduplicators is disclosed. The data segment is received. A first deduplicator is operated to to determine whether the incoming data segment is a duplicate based on first information available to the first deduplicator regarding stored data segments that are stored in a memory. A second deduplicator is selectively operated to determine whether the incoming data segment is a duplicate based on second information available to the second deduplicator; wherein the selective operation of the second deduplicator depends on the determination made by the first deduplicator.