Virtualized Compute Cluster Offloading Storage Network Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage system performance degrades as data access request loads increase, and existing solutions like interconnecting storage systems or proxy caching do not fully address the need to efficiently distribute processing resources and network loads.
Innovation Solution
Employing virtualized compute cluster clients as execution engines for a portion of the storage operating system, where an N-module is ported from a storage system node to a client to utilize available processing bandwidth, thereby distributing storage architecture and offloading network processing loads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If storage systems are interconnected to form a storage system cluster, then the processing load is distributed among nodes, but the device complexity increases
Solution Approach 1:
A virtualization layer is introduced as an intermediary between the storage nodes and compute clients. This virtualization layer abstracts the complex distributed storage architecture, presenting a simplified interface while enabling load distribution across multiple storage nodes without requiring complex configuration at the client level.
Solution Approach 2:
The storage system is segmented into independent virtualized storage nodes that can be dynamically allocated and managed. Each node operates semi-independently, allowing the system to scale by adding individual nodes rather than reconfiguring the entire system, thus managing complexity while improving productivity.
2Productivity
If proxy caching systems are deployed to offload storage processing, then network processing loads on storage nodes are reduced, but the device complexity increases
Solution Approach 1:
The virtualized storage nodes are designed to perform multiple functions including data storage, data processing, and caching operations. By making storage nodes universal and multi-functional, the system eliminates the need for separate proxy caching devices, reducing overall device complexity while maintaining the benefit of offloaded processing.
3Productivity
If virtual machine instances are ported to compute cluster clients, then underutilized client processing power is leveraged, but the device complexity increases
Solution Approach 1:
The virtualized storage system implements self-service capabilities where storage nodes automatically discover available compute resources in the virtualized environment and dynamically allocate processing tasks to underutilized clients. This automation reduces the complexity of manual virtualization management while maximizing resource utilization.
Solution Approach 2:
The system incorporates feedback mechanisms that continuously monitor resource utilization across storage nodes and compute clients. Based on this feedback, the system dynamically adjusts task allocation to balance loads and utilize underutilized client processing power without requiring complex manual intervention.
Data Source
AI summary
A system and method employs one or more clients of a virtualized compute cluster as an execution engine for a portion of a storage operating system implemented as a virtual machine on a storage system node of a storage system cluster. If there is processing bandwidth of a client that is not fully utilized and the load on the storage system node is high, the portion of the storage operating system is ported to the client of the compute cluster in a manner that externally distributes the storage architecture from the storage system cluster. Advantageously, the processing performance of the storage system cluster is improved by, among other things, offloading some of the network processing load from the storage system node.