Sharded Virtual Disk I/O Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for achieving high-performance I/O operations in virtualization systems are limited by the need for explicit reconfiguration of storage systems, which is often inconvenient or impossible, especially when dealing with large data volumes and legacy systems, and do not effectively exploit parallelism across multiple storage hardware.
Innovation Solution
The implementation of dynamic creation and maintenance of sharded virtual storage areas, where a virtual disk is apportioned into non-overlapping shards, each handled by a dedicated shard controller, allowing for real-time optimization of I/O operations across multiple CPU cores and nodes without requiring modifications to the operating system or application code.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is physically distributed across multiple storage drive portions (sharding), then I/O performance is improved through parallelism, but system complexity increases due to required reconfiguration of operating system and application
Solution Approach 1:
A virtualization layer is introduced as an intermediary between the application and physical storage drives. This virtualization layer handles the complexity of data distribution and sharding management, allowing the application to interact with a simplified virtual storage interface while benefiting from parallel I/O operations across multiple physical drives.
Solution Approach 2:
The storage system is segmented into multiple independent storage units or shards that can operate in parallel. Each shard is assigned to a separate storage drive, enabling concurrent I/O operations. The segmentation is managed through virtualization rather than requiring application-level reconfiguration.
2Adaptability or versatility
If data is migrated between nodes in virtualization systems, then flexibility and adaptability are improved, but performance degrades due to heavy load on computing infrastructure when moving large data volumes
Solution Approach 1:
Data migration and movement operations are extracted from the compute nodes and offloaded to dedicated storage nodes or storage area networks. This separation allows compute nodes to maintain flexibility and adaptability for data operations without bearing the heavy load of physical data movement, as storage nodes handle the actual data transfer.
Solution Approach 2:
The system transitions from a single-node architecture to a multi-dimensional distributed architecture where data can be accessed and migrated across multiple nodes simultaneously. This dimensional expansion enables parallel data operations, reducing the load on any single computing infrastructure component while maintaining migration flexibility.
3Reliability
If legacy systems with fixed binary operating systems are used, then system stability is maintained, but I/O performance is limited due to inability to reconfigure storage apportionment
Solution Approach 1:
A virtualization intermediary layer is deployed between the legacy operating system and physical storage resources. This layer provides the reconfiguration and sharding capabilities needed for high-performance I/O operations while the legacy operating system continues to operate unchanged, maintaining system stability without sacrificing performance.
Solution Approach 2:
The virtualization layer serves multiple functions simultaneously: it maintains compatibility with legacy operating systems, enables advanced storage features like sharding and parallel I/O, and provides a unified interface for both legacy and modern applications. This multi-functionality allows legacy systems to achieve modern performance levels without modification.
Data Source
AI summary
Methods, systems, and computer program products for importing a workload from an external system into a virtualization system. A virtual disk (vDisk) is created based on analysis of the data layout of the workload. The vDisk is sharded into a plurality of non-overlapping shards. Independently executing shard controllers are assigned to the non-overlapping shards. On an ongoing basis, the plurality of shard controllers manage respective shards of the plurality of shards. I/O activity on at least some individual ones of the plurality of shard controllers is monitored on an ongoing basis. Two or more shard controllers can be merged into a single shard controller when the I/O activity on the two or more shard controllers is measured to be below a threshold. A single shard controller can be split into two or more shard controllers when I/O activity on the single shard controller is measured to be above a threshold.