Cross-Data-Center File Deduplication for Primary and Backup Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for reducing primary and backup storage are inefficient in handling duplicate and similar data files across different data centers, leading to inadequate storage space utilization and increased costs.
Innovation Solution
A method involving dividing files into blocks, generating hash values, identifying similar files across different storage units, and copying them to a single centralized storage unit while maintaining links to the original locations, thereby leveraging data deduplication techniques to reduce storage space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data de-duplication is applied within a single data centre, then storage space is reduced for that data centre, but duplicate files across different data centres are not efficiently handled
Solution Approach 1:
The patent merges multiple data centre storage systems into a unified de-duplication framework. By establishing a centralized de-duplication manager that coordinates across data centres and using common hash algorithms, the system combines previously isolated de-duplication efforts into a unified approach that handles duplicate files across different data centres efficiently.
Solution Approach 2:
The patent creates a universal de-duplication system that functions across multiple data centres. The common hash algorithm and standardized file identification methods enable the same de-duplication technique to work universally across different storage locations, making the system multi-functional rather than location-specific.
2Reliability
If multiple secondary storage clusters are used to protect one data centre, then backup capacity increases, but storage efficiency decreases due to repeated storage of similar files
Solution Approach 1:
The patent uses selective copying where only unique file portions are replicated to backup storage. By identifying duplicate files through hash comparison and copying only the unique portions to secondary storage clusters, the system maintains data protection while avoiding redundant storage of identical file copies across multiple backup locations.
3Quantity of substance
If compression and encryption are applied to backup data, then storage efficiency improves, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary de-duplication actions before compression and encryption. By first identifying and eliminating duplicate files through hash comparison, then applying compression and encryption only to the reduced unique data set, the system reduces the amount of data requiring time-consuming processing while maintaining security and efficiency benefits.
Data Source
AI summary
A system and method identifies target device file matches of a source file based on file size and hash values. The method includes storing a first source file hash value of the first source file, wherein the first source file hash value was generated by applying a hash function to the first source file, appending a first source file size to the first source file hash value to generate a first appended source file hash value, scanning a target file dataset comprising N target files, wherein the target file dataset is on a target device. Scanning includes determining a respective file size for each of the N target files in the target dataset, determining whether each of the N target files is a size-matching target file, wherein a size-matching target file is any target file of the N target files that matches the first source file size, and generating a respective target file hash value for each size-matching target file by applying the hash function to each size-matching target file.