Backup Stream De-duplication Using Extent Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data de-duplication techniques struggle to recognize and eliminate redundant data files across different storage devices, leading to inefficiencies in de-duplication rates due to varying file locations and lack of boundary recognition in image-based backups.
Innovation Solution
The method involves processing a backup stream to access extent mapping, identifying extent groups, and using this information to optimize de-duplication by locating and removing redundant data files, thereby enhancing the de-duplication rate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If fixed size algorithm based de-duplication technique is used, then data blocks can be processed systematically, but identical files at different locations cannot be recognized
Solution Approach 1:
The patent segments the backup stream into fixed-size data blocks and processes them systematically through hash functions. Each data block is independently hashed and compared against the lookup table, enabling efficient systematic processing while maintaining the ability to identify identical content regardless of position.
Solution Approach 2:
The patent uses hash values as copies of the actual data blocks for comparison purposes. Instead of comparing entire files or relying on location information, it creates hash copies that represent the content essence, allowing identical files at different locations to be recognized through their matching hash values.
2Productivity
If de-duplication techniques utilize file boundaries, then redundant data files can be removed, but image-based backups cannot determine data file boundaries
Solution Approach 1:
The patent extracts the essential identifying feature (hash value) from each data block without requiring file boundary information. By taking out the content signature rather than relying on structural boundaries, it enables redundant data removal even in image-based backups where file boundaries are indeterminate.
Solution Approach 2:
The patent changes the parameter used for identification from location-based (file boundaries) to content-based (hash values). This parameter transformation allows the system to work effectively with image-based backups where traditional file boundary detection is not applicable.
3Quantity of substance
If conventional de-duplication techniques are used, then storage space can be optimized, but de-duplication rate is reduced due to varying file locations
Solution Approach 1:
The patent creates a universal hash-based identification system that works across all data blocks regardless of their source location or file boundary context. This multi-functional approach allows the same hash comparison mechanism to effectively identify duplicates whether they come from identical files at different locations or from image-based backups without clear boundaries.
Data Source
AI summary
A method and apparatus for optimizing a de-duplication rate for backup streams is described. In one embodiment, the method for optimizing data de-duplication using an extent mapping of a backup stream includes processing a backup stream to access an extent mapping associated with a plurality of data files, wherein the plurality of the data files are arranged within the backup stream and examining the extent mapping to identify at least one extent group within the backup stream, wherein the plurality of the data files are de-duplicated using at least one location of the at least one extent group.