SAN Compute Resource Scheduling via Idle Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage area networks (SANs) experience significant underutilization of computing resources due to idle periods in both hosts and storage arrays, leading to a non-trivial amount of unused resources.
Innovation Solution
A method for scheduling and executing workloads on nodes predicted to be idle, utilizing a predictive model and resource monitoring to dynamically allocate tasks to nodes with available resources, and implementing a distributed resource assignment and scheduling system with APIs for efficient resource management and communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If storage arrays and hosts are deployed with additional resources provisioned for future growth and peak workloads, then system reliability and future adaptability are improved, but resource utilization efficiency deteriorates
Solution Approach 1:
The patent enables storage array computing resources to perform dual functions: their primary storage function and secondary distributed computing function. The system allows storage arrays to dynamically share their idle CPU and memory resources with compute nodes, transforming single-function hardware into multi-functional systems that improve overall resource utilization without compromising storage reliability
Solution Approach 2:
The system implements dynamic resource allocation where storage arrays can adjust their computing resource availability in real-time based on workload conditions. The compute nodes dynamically discover and utilize available resources from storage arrays, and resources can be added or removed from the pool based on changing system conditions, making the resource allocation flexible and adaptive
2Productivity
If idle computing resources are shared across the SAN network, then resource utilization efficiency is improved, but system complexity increases
Solution Approach 1:
The system implements self-service mechanisms where compute nodes autonomously discover available computing resources on storage arrays without requiring manual configuration or complex centralized management. The storage arrays themselves publish their available resources, and compute nodes automatically subscribe and utilize these resources, reducing the need for complex external orchestration
Solution Approach 2:
The patent introduces a resource manager component that acts as an intermediary between storage arrays and compute nodes. This intermediary handles resource registration, capability matching, and allocation coordination, simplifying the overall system architecture by centralizing management functions while maintaining distributed resource utilization
3Measurement precision
If predictive models are used to identify idle resources, then scheduling accuracy is improved, but computational overhead increases
Solution Approach 1:
The system performs preliminary actions by having storage arrays continuously monitor and report their resource utilization status and idle periods in advance. Predictive models analyze historical utilization patterns to forecast future idle windows, allowing compute nodes to proactively schedule workloads during predicted idle periods without requiring complex real-time analysis or excessive computational overhead during actual scheduling
Data Source
AI summary
Embodiments are described for a executing a processing job using one or more nodes of a storage area network using computing resources on the SAN that are predicted to be idle. A predictive model is generated by monitoring idle states of resources of nodes of the SAN and using machine learning to build the predictive model. A scheduler executes jobs on one or more nodes of the SAN with sufficient predicted idle resources to process the job, in accordance with resource requirements and job attributes in a manifest of the job. If a job cannot be completed during a window of time that the necessary resources are predicted to be idle, or if one or more resources become unavailable, the job can be paused and resumed, migrated to another node, or restarted at a later time when the required resources to complete the job are predicted to be idle.