Storage Virtualizer for Flushing Data in Virtual Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In virtualized computing systems, the performance of applications is limited by the slower speed of Hard Disk Drives (HDDs) and Solid State Drives (SSDs) during data flushing, especially due to random writes to backend storage, which impede processing efficiency.
Innovation Solution
A method and system for efficiently flushing data by generating a mapping of application writes, sorting virtual disks based on write offsets, canceling duplicates, merging contiguous writes, and merging epochs prior to flushing, while also performing inline deduplication and compression to reduce Input/Output operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is flushed to backend HDDs and SSDs during application execution, then data persistence is achieved, but application performance is severely limited by the slower speed of backend storage compared to cache
Solution Approach 1:
The patent performs preliminary actions by sorting write operations by offset before flushing to backend storage, merging contiguous writes into sequential operations, and pre-processing data to eliminate random writes. This preparation work is done in advance of the actual flush operation, allowing the backend storage to receive optimized write patterns that improve performance while maintaining data persistence.
Solution Approach 2:
The patent introduces an intermediary processing layer between the application/cache and backend storage. This intermediary sorts writes, merges contiguous operations, eliminates duplicates, and reorganizes data before it reaches the backend HDDs/SSDs. This mediator transforms random cache writes into sequential storage writes, improving backend performance without affecting application visibility.
2Productivity
If random writes are performed to backend storage during data flush, then application writes are captured, but flushing performance is impeded because HDDs are much faster at sequential writes than random ones
Solution Approach 1:
The system performs preliminary sorting of write operations by offset before flushing to backend storage. By organizing writes in sequential order based on their target offsets, the system converts random write patterns into sequential write patterns that HDDs can handle efficiently, thereby improving flushing speed while maintaining complete write capture.
Solution Approach 2:
The patent merges multiple contiguous writes into single sequential write operations. By combining adjacent writes that target contiguous regions of storage, the system reduces the total number of I/O operations and transforms multiple random writes into fewer sequential writes, significantly improving flushing performance.
3Speed
If cache space is filled during application execution, then fast writing is achieved, but older data must be flushed to slower backend storage which limits application performance
Solution Approach 1:
The system performs preliminary processing of data before flushing to backend storage, including sorting by offset, merging contiguous writes, and eliminating duplicates. This pre-processing reduces the amount of data that needs to be flushed and optimizes the write pattern, thereby reducing flush time while maintaining the benefit of fast cache writing.
Solution Approach 2:
The patent identifies and eliminates duplicate writes before flushing to backend storage. By detecting redundant write operations and discarding them, the system reduces the total volume of data that must be persisted, thereby reducing flush time while maintaining data integrity through selective recovery of unique writes.
Data Source
AI summary
A method for flushing data in a virtual computing environment is provided. The method includes writing application output from an application spanning one or more virtual machines to a cache, wherein each virtual machine is implemented using one or more compute nodes and the cache is implemented in storage associated with the one or more compute nodes. In an effort to flush data associated with application writes more efficiently from the cache to a local backing store or one or more data nodes, the method may include generating a mapping of each of the plurality of application writes. The method may further include sorting sets of one or more virtual disks based upon an offset of each application write as indicated in the mapping. In addition, a storage virtualizer may cancel duplicate application writes, merge multiple contiguous application writes, and merge multiple epochs prior to flushing the data.