Converged Storage Network Controller for Virtualized Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage and networking solutions for scale-out applications face challenges such as high latency, inefficient data transfer, and inconsistent performance due to complex IO operations, static resource allocation, and the need for expensive hardware, particularly in hypervisor and bare metal solutions, which result in poor performance and scalability.
Innovation Solution
The implementation of a converged networking and storage system that virtualizes storage resources, allowing for dynamic resource allocation, efficient data transfer, and improved scalability by using NVMe over Ethernet (NVMeoE) with a Burst Transmission Protocol (BTP) for guaranteed delivery and error recovery, and a converged storage and network controller that combines initiator, target storage, and network functions into a single data and control path.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If virtual machine solutions are used for scale-out applications, then resource allocation flexibility is improved, but IO performance and consistency deteriorate
Solution Approach 1:
The system segments storage resources into virtual storage pools that can be dynamically allocated to multiple virtual machines. Each VM receives dedicated storage I/O paths through virtualization, ensuring consistent performance while maintaining flexibility in resource distribution across the cluster.
Solution Approach 2:
A storage virtualization layer acts as an intermediary between physical storage resources and virtual machines. This layer abstracts the complexity of resource management, providing consistent I/O performance through optimized data paths while enabling flexible resource allocation and dynamic provisioning.
2Productivity
If bare metal solutions are used for scale-out applications, then IO performance is improved, but resource allocation flexibility and scalability deteriorate
Solution Approach 1:
The system creates a universal storage platform that combines the raw performance of bare metal with the flexibility of virtualization. The converged network/storage controller provides multi-functional capabilities, enabling both high-performance direct I/O paths and flexible virtualized resource allocation within the same infrastructure.
Solution Approach 2:
The system implements dynamic resource allocation where storage resources can be flexibly assigned and reassigned based on workload demands. The virtualization layer allows real-time adjustment of resource allocation without sacrificing I/O performance, enabling scalable deployment across multiple nodes.
3Reliability
If complex IO operations are used in current storage systems, then data transfer reliability is improved, but latency and performance consistency deteriorate
Solution Approach 1:
The system performs preliminary actions by pre-establishing optimized I/O paths and caching mechanisms before data transfer operations. The converged controller prepares data buffers and transmission channels in advance, reducing latency while maintaining reliability through pre-configured error handling and data integrity checks.
Solution Approach 2:
The system extracts and separates critical I/O operations from complex software stacks, moving them to hardware-level processing in the converged controller. This extraction eliminates unnecessary software overhead and latency-introducing operations while preserving essential reliability functions through dedicated hardware validation.
4Productivity
If expensive hardware is used to achieve high performance, then IO performance is improved, but cost and scalability deteriorate
Solution Approach 1:
The system uses software-based virtualization to create copies of high-performance I/O paths, allowing multiple virtual machines to access storage resources with near-physical performance characteristics. This eliminates the need for expensive dedicated hardware per VM while maintaining high performance through efficient resource sharing and isolation.
Solution Approach 2:
The system changes the parameters of hardware utilization by implementing intelligent resource pooling and dynamic allocation. The converged controller optimizes hardware performance parameters through software management, achieving high I/O performance with commodity hardware and enabling scalable deployment without proportional increases in hardware cost.
Data Source
AI summary
Provided herein are methods and systems for improved storage strategies for use of collections of storage resources, such as solid state drives, including in connection with a converged networking and storage node that may be used for virtualization of a collection of physically attached and/or network-connected storage resources.