Unified Change Log for Network Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication operations in network storage systems face inefficiencies in change logging during multi-phase backup processes, leading to unnecessary complexity and storage overhead due to the generation of multiple entries for data blocks shared among files.
Innovation Solution
A two-phase transfer method is implemented, where the first phase involves transferring data blocks and generating a data log with fingerprints, and the second phase transfers metadata associating data blocks with files, merging these logs to create a change log that efficiently tracks file relationships and duplicates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional deduplication operations generate separate logs for data blocks and metadata, then complete tracking of file relationships is achieved, but redundant entries and storage overhead increase
Solution Approach 1:
The patent merges the data block log and metadata log into a single unified change log. The change log manager combines entries from both logs, eliminating redundant information and storing only unique file change records. This reduces storage overhead while maintaining complete tracking of file relationships through the integrated log structure.
Solution Approach 2:
The unified change log serves multiple functions simultaneously: it tracks data block changes, records metadata associations, and identifies duplicate files. By making the change log multi-functional, the system eliminates the need for separate specialized logs, reducing overall storage requirements while maintaining comprehensive change tracking capabilities.
2Loss of information
If separate logs are maintained for data blocks and metadata, then detailed tracking information is preserved, but processing complexity increases
Solution Approach 1:
The patent combines separate data block logging and metadata logging operations into a single unified change logging process. The change log manager integrates both types of information into one standardized log structure, simplifying log management operations while preserving all necessary tracking information about file changes and relationships.
Solution Approach 2:
The unified change log is structured with distinct entry types for data block changes and metadata associations, allowing the system to process and manage different kinds of information separately within a unified framework. This segmented structure within unity reduces processing complexity by providing clear organization while maintaining comprehensive tracking.
3Reliability
If duplicate data blocks are stored separately, then data integrity is maintained, but storage efficiency decreases
Solution Approach 1:
The patent extracts duplicate data blocks from multiple files and stores only a single copy in the backup storage. The change log tracks which files reference each unique data block, allowing the system to maintain data integrity through reference tracking while eliminating redundant storage of identical data blocks across multiple files.
Data Source
AI summary
A method performed in a network storage system, the method including receiving a plurality of data blocks at a secondary storage subsystem from a primary storage subsystem, generating a first log that includes a first plurality of entries, one entry for each of the data blocks, in which each entry of the first plurality of entries includes a name for a respective data block and a fingerprint of the respective data block, receiving metadata at the secondary storage subsystem from the primary storage subsystem, the metadata describing relationships between the plurality of blocks and a plurality of files, generating a second log that includes a second plurality of entries, and merging the first log with the second log to generate a change log.