Block-Hash File Consolidation Across Data Centers for Storage Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for reducing primary and backup storage are inefficient in handling duplicate and similar data files across different data centers, leading to unnecessary space occupation and inadequate de-duplication.
Innovation Solution
A method involving dividing files into blocks, generating hash values, identifying similar files across different storage units, and copying them to a central storage unit while maintaining links in the source, thereby reducing primary and backup storage through data deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data de-duplication is applied within a single data centre using secondary storage protection scope, then backup storage capacity is reduced for that specific data centre, but duplicate data files stored on different data centres are not de-duped efficiently and continue to occupy more space
Solution Approach 1:
The patent implements a universal de-duplication system that functions across multiple data centres by introducing a global de-duplication manager that coordinates de-duplication operations across different storage clusters. This allows the same de-duplication technology to serve both single-data-centre and multi-data-centre scenarios, eliminating duplicate files enterprise-wide rather than being limited to individual data centre boundaries.
Solution Approach 2:
The patent transitions from a single-data-centre de-duplication approach to a multi-data-centre approach by adding the dimension of cross-data-centre coordination. This is achieved through a hierarchical architecture where global de-duplication managers at the enterprise level oversee multiple local de-duplication managers at individual data centre levels, enabling de-duplication to operate across the additional organizational dimension of multiple data centres.
2Reliability
If multiple secondary storage clusters are used to protect one data centre, then storage protection scope is expanded, but the complexity of managing de-duplication across these clusters increases
Solution Approach 1:
The patent segments the de-duplication management system into hierarchical levels: global de-duplication managers at the enterprise level and local de-duplication managers at individual storage cluster levels. This segmentation allows each manager to handle de-duplication within its specific scope, reducing the complexity that would arise from a monolithic management approach while maintaining coordinated operation across multiple storage clusters.
Solution Approach 2:
The patent introduces global de-duplication managers as intermediary components that coordinate between multiple local de-duplication managers and the enterprise-wide storage system. These intermediaries abstract the complexity of cross-cluster coordination, allowing local managers to operate independently while still achieving global de-duplication goals, thus reducing overall system complexity.
3Reliability
If duplicate data files are stored on different data centres, then data availability is maintained across locations, but storage space is wasted due to inability to de-duplicate across data centre boundaries
Solution Approach 1:
The patent merges de-duplication operations across previously isolated data centre boundaries by implementing a unified global de-duplication management system. This allows duplicate data files stored on different data centres to be identified and consolidated, eliminating redundant storage while maintaining data availability through the distributed architecture that preserves local access capabilities.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
A method of data management in a data storage system including a plurality of data storage units, the method includes dividing each file in the data storage system into a plurality of blocks having a common size. The method further includes generating a hash value for each block using a common hash algorithm. The method further includes identifying a plurality of similar files in different data storage units, based on a comparison of the hashes for each file. The method further includes copying one or more of the identified similar files, such that similar files are stored in a single data storage unit and for one or more of the copied files, generating a link to a new location of the file. The method provides an efficient, effective, and adequate reduction of the primary storage as well as the secondary storage space.