Cohort Management for Data Deduplication Version Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems are inefficient in managing version updates, leading to increased storage space usage due to duplicate data blocks across multiple copies of the same data, as they fail to optimally deduplicate updated versions of files and documents across different hosts and virtual machines.
Innovation Solution
An apparatus and method for improved cohort management in data deduplication, which includes a baseline module for storing referrer links, a signpost module for pointing to updated data blocks, and a cohort update module that replaces baseline referrer links with updated referrer links in a deduplicating storage device, leveraging metadata and hash comparisons to efficiently manage version updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data deduplication systems store separate copies of updated data blocks across multiple hosts, then each host can access its own version independently, but storage space is wasted due to duplicate data blocks
Solution Approach 1:
The patent merges multiple versions of the same data block into a single stored copy by using version identifiers and comparison logic. When an updated data block is written, the system compares it with existing versions and stores it only if it's genuinely new, otherwise it updates references to the existing copy. This combining approach eliminates duplicate storage while maintaining version accessibility through reference management.
Solution Approach 2:
The patent introduces an intermediary deduplication layer between storage and hosts that manages version references. This intermediary maintains metadata about data block versions and provides a unified interface for accessing different versions, allowing multiple hosts to reference the same physical storage location while accessing different logical versions of data.
2Loss of information
If the system stores all version updates of data blocks, then complete version history is maintained, but storage efficiency decreases due to duplicate blocks
Solution Approach 1:
The patent uses metadata copying rather than physical data copying to maintain version history. Instead of storing multiple physical copies of data blocks, it creates and stores lightweight metadata records that reference the actual data blocks. These metadata records contain version information and references, allowing the system to track complete version history while consuming minimal storage space.
Solution Approach 2:
The patent segments the storage system into two distinct parts: physical data block storage and metadata storage. The physical storage holds unique data blocks, while the metadata storage maintains version information and references. This segmentation allows the system to preserve complete version history in the metadata layer without duplicating the actual data blocks in the physical storage layer.
3Quantity of substance
If the system performs comprehensive deduplication checks across all data blocks, then deduplication efficiency is maximized, but processing time and performance are reduced
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing hash values or fingerprints of data blocks when they are first written. When subsequent writes occur, the system quickly compares these pre-stored hashes with incoming data block hashes to determine if deduplication is possible. This preliminary preparation eliminates the need for comprehensive byte-by-byte comparison during normal operations, significantly improving processing speed while maintaining deduplication efficiency.
Data Source
AI summary
An apparatus for improved version updates in data deduplication includes a baseline module that stores a baseline referrer link in a first cohort of baseline referrer links pointing to a first location in storage where a baseline version data block is stored. The apparatus includes a signpost module that stores to metadata a signpost that points to an updated referrer link in a second cohort of updated referrer links that points to a second location in the deduplicating storage device where an updated version data block is stored in response to the updated version data block being generated. The apparatus includes a cohort update module that replaces the baseline referrer link with the updated referrer link in response to determining that an additional updated version data block matches the updated version data block pointed to by the signpost. A method and a system implement functions of the apparatus.