Virtual Disk De-Duplication Using Global Block Message Digests
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies and high costs due to the overwhelming amount of data stored, particularly with the advent of virtualization, where multiple applications on a single server lead to duplicate data being written across various storage platforms, which are not optimally addressed by current de-duplication techniques.
Innovation Solution
Implementing a storage platform with de-duplication features that calculate unique message digests for each block of data, preventing duplicates by consulting a hash table before writing, and using a system virtual disk for all data blocks, enabling de-duplication across the entire platform, including virtual disks and public cloud storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data is stored locally at each computer server, then data storage is straightforward, but storage manageability deteriorates due to data being stored in many different places
Solution Approach 1:
The patent merges storage operations across multiple virtual disks into a unified de-duplication system. The system consolidates data from multiple sources by calculating message digests and storing only unique blocks, combining what was previously scattered across individual server storages into a centralized de-duplicated repository that maintains simplicity while improving manageability.
2Ease of operation
If a central storage node is used, then storage manageability is improved, but the system becomes overwhelmed with the sheer number of server applications accessing disks
Solution Approach 1:
The patent extracts the de-duplication function from the central storage node and implements it at the data block level across all virtual disks. By calculating message digests and identifying duplicate blocks before storage, the system removes redundant data writes at the source, reducing the burden on the central storage platform while maintaining centralized manageability.
Solution Approach 2:
The system performs preliminary de-duplication actions by calculating message digests and checking for existing blocks before data is written to storage. This preliminary identification and elimination of duplicates prevents unnecessary storage operations, reducing the workload on the central storage platform before data arrives.
3Loss of substance
If de-duplication is implemented at the virtual disk level, then duplicate removal is achieved, but de-duplication does not occur across the entire storage platform
Solution Approach 1:
The patent implements a universal de-duplication system that functions across all virtual disks in the storage platform. The message digest calculation and duplicate detection mechanism is applied universally to data blocks from any virtual disk, enabling the system to identify and eliminate duplicates across the entire platform rather than being limited to individual disk boundaries.
Data Source
AI summary
In order to avoid writing duplicates of blocks of data into a storage platform, any virtual disk within the storage platform may have a de-duplication feature enabled. Or, all virtual disks have this feature enabled. For virtual disks with de-duplication enabled, a unique message digest is calculated for every block of data written to that virtual disk. Upon a write, these message digests are consulted in order to determine if a particular block of data has already been written, if so, it is not written again, and if not, it is written. All de-duplication virtual disks are written to a single system virtual disk within the storage platform. De-duplication occurs over the entire storage platform and over all its virtual disks because all message digests are consulted before a write is performed for any virtual disk. A read for a de-duplication virtual desk reads from the system virtual disk.