Global De-duplication Across Shared Storage Architectures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data de-duplication technologies are limited to individual storage systems and fail to globally reduce redundant data across multiple storage systems, even with snapshot technologies like copy-on-write and WAFL file systems.
Innovation Solution
Implementing a de-duplication client on each storage system that maintains a hash table of data and communicates with a de-duplication server to identify and eliminate duplicate data across multiple storage systems by replacing redundant data with pointers to a single instance, using hash tables and inode modifications in WAFL file systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If snapshot technologies like copy-on-write and WAFL file systems are used to share unchanged data within a single storage system, then storage requirements are reduced, but redundant data remains across multiple storage systems
Solution Approach 1:
The patent merges multiple storage systems into a unified de-duplication domain by implementing a centralized de-duplication server that coordinates across all storage systems. The server maintains a global registry of data blocks and their locations, enabling cross-system data sharing. This combines the benefits of local copy-on-write snapshots with a centralized coordination mechanism to achieve global de-duplication.
Solution Approach 2:
The de-duplication server acts as an intermediary between storage systems, maintaining a global registry that maps data block hashes to their physical locations across all systems. When a storage system needs to de-duplicate data, it queries the server's registry rather than performing local de-duplication only, enabling efficient cross-system data sharing without requiring direct peer-to-peer communication.
2Productivity
If traditional copy-on-write snapshot technology is used, then data sharing within a storage system is improved, but redundant data cannot be eliminated globally across multiple storage systems
Solution Approach 1:
The de-duplication server implements a feedback mechanism where storage systems report their data block locations and hashes to the server's registry. When a storage system creates a snapshot or writes new data, it updates the registry with the new block information. This feedback loop enables the system to learn about data locations across all storage systems and make intelligent de-duplication decisions.
Solution Approach 2:
The system performs preliminary actions by pre-establishing a global registry of data blocks before de-duplication operations begin. The registry is continuously updated as storage systems write data, so when a de-duplication operation is needed, the information about existing data locations is already available, enabling rapid cross-system de-duplication without requiring full scans of all storage systems.
3Reliability
If data is stored in multiple storage systems without global de-duplication, then data availability and redundancy are improved, but storage capacity is wasted due to duplicate data
Solution Approach 1:
The de-duplication system is dynamic rather than static. The global registry is continuously updated as data is written, deleted, or moved across storage systems. When data is deleted from one system, the registry is updated to reflect this, allowing other systems to reclaim their copies. This dynamic maintenance of the registry enables the system to adapt to changing data locations while maintaining de-duplication benefits.
Solution Approach 2:
The system discards redundant data copies by replacing them with pointers to the original data block stored in a different storage system. The de-duplication server tracks which blocks are duplicates and manages the transition from local copies to shared references. When data is deleted from one system, the server recovers the space by updating the registry to indicate the block is no longer needed locally, allowing other systems to reclaim their copies.
Data Source
AI summary
Redundant data is globally de-duplicated across a shared architecture that includes a plurality of storage systems. The storage systems implement copy-on-write or WAFL to generate snapshots of original data. Each storage system includes a de-duplication client to identify and reduce redundant original and/or snapshot data on the storage system. Each de-duplication client can de-duplicate a digital sequence by breaking the sequence into blocks and identifying redundant blocks already stored in the shared architecture. Identifying redundant blocks may include hashing each block and comparing the hash to a local and/or master hash table containing hashes of existing data. Once identified, redundant data previously stored is deleted (e.g., post-process de-duplication), or redundant data is not stored to begin with (e.g., inline de-duplication). In both cases, pointers to shared data blocks can be used to reassemble the digital sequence where one or more blocks were deleted or not stored on the storage system.