Shared Folder Migration for De-duplication Block Size Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed file systems, the inefficiency in deleting duplicate data across different storage systems with varying de-duplication block sizes leads to suboptimal storage space utilization and de-duplication rates, as each system maintains its own hash table and lacks awareness of global data patterns, resulting in missed opportunities for data deduplication.
Innovation Solution
A method that determines the target data block size for a shared folder, calculates the de-duplication rate, and migrates the folder to a storage system with a matching de-duplication block size, optimizing de-duplication efficiency by retaining only high-probability entries in the hash table and improving resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each storage system maintains its own de-duplication hash table, then the system can operate independently, but the de-duplication rate is reduced due to lack of global data awareness
Solution Approach 1:
The patent merges the de-duplication hash tables from multiple storage systems into a unified global hash table. This allows the system to maintain independence at the storage system level while achieving global de-duplication through the combined hash table, thereby resolving the contradiction between system independence and de-duplication rate.
Solution Approach 2:
The unified hash table serves multiple storage systems simultaneously, providing a universal de-duplication mechanism that works across different storage systems with varying block sizes. This multi-functional approach enables global de-duplication while maintaining the independence of individual storage systems.
2Adaptability or versatility
If storage systems use different de-duplication block sizes, then they can be optimized for different applications, but data deduplication efficiency is lost across systems
Solution Approach 1:
The patent changes the parameter of hash table organization to accommodate different block sizes. By organizing the unified hash table to handle variable block sizes from different storage systems, it maintains application-specific optimization while enabling cross-system deduplication efficiency.
3Productivity
If a unified hash table is implemented across storage systems, then global de-duplication is achieved, but system complexity increases
Solution Approach 1:
The patent segments the unified hash table into portions corresponding to different storage systems or block size ranges. This segmentation allows for more manageable implementation and operation of the global hash table while still achieving unified de-duplication across all systems.
Data Source
AI summary
Techniques involve determining, in response to acquiring a target block size for a shared folder of a first file system in a file system set, a shared folder de-duplication rate. The techniques further involve acquiring a de-duplication block size set supported by a storage system set for the file system set in response to determining that the de-duplication rate is less than a threshold de-duplication rate. The techniques further involve determining whether the de-duplication block size set includes a de-duplication block size that is the same as the target block size. The techniques further involve migrating the shared folder to a second file system in response to the de-duplication block size set including the de-duplication block size that is the same as the target block size. Accordingly, there is improved efficiency in deleting duplicate data, increased amount of deleted duplicate data, saved storage space and improved resource utilization.