Disk Image De-duplication via Shared Storage References
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud environments, data redundancy in virtual machine disks leads to increased storage space consumption and I/O bottlenecks due to duplicate files, which can degrade server performance.
Innovation Solution
A data processing system with parallel disk emulators and a de-duplicator that identifies and replaces duplicate data with references, using an additional disk emulator to store shared data, allowing for atomic operations and minimizing performance impact during runtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored multiple times in virtual machine disks, then storage capacity is sufficient, but storage space consumption increases and I/O performance degrades
Solution Approach 1:
The patent uses copying by creating references to duplicate data blocks in a shared storage location instead of storing actual copies of the data multiple times. When a virtual machine needs data, the system copies the reference pointer to the shared storage rather than copying the data itself, thereby reducing storage space consumption while maintaining data accessibility.
Solution Approach 2:
The patent introduces a shared storage location as an intermediary between multiple virtual machines and their data blocks. This intermediary stores both the actual data blocks and reference pointers to them, allowing multiple VMs to access the same data without duplicating it, thus resolving the contradiction between storage capacity and space consumption.
2Reliability
If duplicate files are stored in disk caches, then data availability is improved, but I/O utilization increases and becomes a bottleneck
Solution Approach 1:
The patent applies copying at the block level by storing reference pointers to data blocks in the cache rather than storing complete duplicate files. This allows the cache to maintain data availability references while minimizing the amount of actual data written to and read from the I/O subsystem, thereby reducing I/O utilization.
Solution Approach 2:
The patent implements discarding duplicate data blocks from individual VM disk images and recovering them by consolidating them into a shared storage location. This process eliminates redundant data while maintaining access to the data through references, thereby reducing I/O operations required for data retrieval.
3Stability of the object's composition
If cache flushes frequently to access I/O subsystem, then data consistency is maintained, but server performance degrades
Solution Approach 1:
The patent introduces a shared storage location as an intermediary that maintains data consistency across all virtual machines. By storing reference pointers to data blocks in this shared location, the system can maintain data consistency without requiring frequent cache flushes to the I/O subsystem, as the reference structure itself provides consistent access paths.
Solution Approach 2:
The patent uses copying of reference pointers instead of copying actual data during cache operations. This allows the cache to maintain consistency by updating references rather than flushing complete data blocks, thereby reducing the frequency and impact of I/O subsystem access while maintaining data consistency.
4Quantity of substance
If storage space is optimized by removing duplicates, then space consumption decreases, but data retrieval complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the storage structure into separate components: data blocks stored in shared storage and reference pointers stored in VM-specific disk images. This segmentation allows the system to optimize space by removing duplicate data blocks while simplifying data retrieval through the reference pointer structure, which maps VMs to their respective data references without requiring complex search operations.
Data Source
AI summary
The invention relates to a data processing system including at least two disk emulators operating in parallel. Each of the at least two disk emulators emulates a disk subsystem and is associated with a respective file in a file system for any data stored on a physical disk. The data processing system further includes a de-duplicator for de-duplicating the data stored in the respective files associated with the at least two disk emulators. The de-duplicator operates in parallel to the disk emulators and is associated with an additional disk emulator emulating an additional disk subsystem. The additional disk emulator is associated with an additional file in a file system configured to store data shared between the respective disk subsystems emulated by the at least two disk emulators.