Space-Efficient Virtual Disk Format for Storage Reclamation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Virtualization technologies face storage inefficiencies due to initial over-allocation of physical storage space for virtual disks and the accumulation of 'stale' data blocks, which are not timely reused or reclaimed effectively, leading to resource wastage.
Innovation Solution
A space-efficient virtual disk format is implemented, where data is stored in granular units called 'grains' that are dynamically allocated and deallocated, with a defragmentation process that moves unused grains to the end of the disk, allowing for truncation and space reclamation by the virtual machine file system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If physical storage space is allocated for virtual disk at initial creation, then virtual disk is ready for immediate use, but storage space is wasted due to over-allocation
Solution Approach 1:
The virtualization software pre-allocates physical storage space for the virtual disk at creation time, ensuring the virtual disk is immediately ready for use. However, the space is reserved rather than fully committed, allowing the system to later determine actual usage and reclaim unused portions, thus resolving the contradiction between immediate readiness and storage waste.
Solution Approach 2:
The system dynamically changes the allocation status of physical storage space from pre-allocated (reserved) to actually used based on real demand. By monitoring and adjusting the mapping between virtual and physical blocks, the system transforms the static over-allocation into a dynamic parameter that adapts to actual usage, eliminating storage waste while maintaining operational readiness.
2Loss of substance
If virtualization software dynamically allocates physical storage space when needed, then storage efficiency is improved, but storage space is not available immediately upon virtual disk creation
Solution Approach 1:
The system performs preliminary allocation of physical storage space at virtual disk creation, making the virtual disk immediately ready for operations. The allocation is initially reserved rather than fully committed, enabling the system to later transition to dynamic allocation based on actual usage patterns, thus maintaining both immediate readiness and storage efficiency.
Solution Approach 2:
The storage allocation system transitions from a static pre-allocation model to a dynamic model where physical space is allocated based on real-time usage demands. The system continuously monitors virtual disk access patterns and adjusts the mapping between virtual and physical blocks dynamically, ensuring space is available when needed while minimizing waste.
3Ease of operation
If guest operating system deletes a file, then the file is removed from view, but the underlying data blocks remain allocated and cannot be reclaimed
Solution Approach 1:
The system implements a feedback mechanism where the virtualization software continuously monitors the guest operating system's file system operations. When a file is deleted, the feedback loop detects this event and triggers the reclamation of the underlying data blocks by updating the mapping structures, thus converting the operation from a simple deletion to a cascading resource release that eliminates storage waste.
Solution Approach 2:
The system extracts and separates the management of data block allocation from the guest operating system's file system. By maintaining independent mapping structures that track the actual usage of physical blocks, the system can identify and reclaim blocks that are no longer needed, even when the guest OS has deleted files, thus extracting the wasted allocation and making it available for reuse.
4Device complexity
If virtual disk uses large block sizes for storage, then storage management is simplified, but fine-grained allocation and deallocation becomes difficult
Solution Approach 1:
The system segments the virtual disk into a hierarchy of block sizes: large virtual blocks for simplified management and smaller physical blocks for flexible allocation. The mapping structure divides the virtual disk into virtual blocks that can be efficiently managed, while each virtual block is further segmented into multiple physical blocks that can be independently allocated and deallocated, thus resolving the contradiction between management simplicity and allocation flexibility.
Solution Approach 2:
The system implements a nested block structure where large virtual blocks contain multiple smaller physical blocks. This nested organization allows the outer layer (virtual blocks) to provide simplified management with large unit sizes, while the inner layer (physical blocks) enables fine-grained allocation and deallocation. The mapping structure navigates through these nested levels to achieve both management simplicity and allocation flexibility.
Data Source
AI summary
A virtualized computer system employs a virtual disk with a space efficient (SE) format to store data for virtual machines running therein. The SE format allows for defragmentation at a fine-grained level, where unused, stale, and zero blocks are moved to the end of the virtual disk so that the virtual disk may be truncated and space reclaimed by the underlying storage system as part of a special defragmentation process.