GPU Virtual Node Scheduling by Hardware Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Elastic scaling methods in Serverless platforms for heterogeneous computing tasks based on GPU hardware do not consider the utilization rate of various components, leading to cost waste and performance losses due to over-expansion or premature narrowing.
Innovation Solution
A task processing method that determines current state information of initial virtual nodes based on physical computing units, selects candidate virtual nodes based on task type information, and executes tasks through the most suitable target virtual node, allowing for flexible scaling based on hardware utilization rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If elastic scaling is implemented without considering hardware component utilization rates, then scaling speed is improved, but resource waste and performance loss occur
Solution Approach 1:
The patent implements feedback by monitoring the utilization rates of GPU hardware components (CUDA cores, tensor cores, hardware encoders/decoders) and using this information to dynamically adjust virtual node scaling decisions. The scheduling system receives utilization rate information from the GPU monitor and incorporates it into the scheduling strategy, creating a closed-loop feedback mechanism that prevents both over-expansion and premature narrowing while maintaining fast scaling responses.
Solution Approach 2:
The patent changes the scaling parameters from simple request concurrency counts to utilization-rate-based metrics. Instead of scaling solely based on the number of requests, the system now scales based on actual hardware component utilization rates, allowing for more precise control of virtual node expansion and contraction that aligns with real resource consumption patterns.
2Adaptability or versatility
If virtual nodes are scaled based on request concurrency, then scalability is improved, but hardware resource utilization efficiency deteriorates
Solution Approach 1:
The system continuously monitors GPU hardware component utilization rates and feeds this information back to the scheduling strategy. This feedback loop enables the system to adjust virtual node allocation based on actual hardware usage patterns, improving resource utilization efficiency while maintaining the ability to scale up or down as needed.
Solution Approach 2:
The patent transforms the scaling decision parameters from request concurrency-based metrics to hardware utilization rate-based metrics. By using utilization rates of specific hardware components (CUDA cores, tensor cores, encoders/decoders) as the primary scaling parameter, the system achieves both scalability and improved resource utilization efficiency simultaneously.
3Reliability
If GPU instances are expanded to handle peak loads, then service availability is improved, but cost increases due to idle resources during low demand
Solution Approach 1:
The feedback mechanism monitors actual GPU hardware utilization rates and uses this information to dynamically adjust the number of active virtual nodes. During low-demand periods, the system identifies and terminates idle virtual nodes, reducing resource consumption while maintaining service availability during peak loads through rapid virtual node deployment when utilization rates increase.
Solution Approach 2:
The patent implements dynamic scaling where the number of active virtual nodes is continuously adjusted based on real-time hardware utilization rates. The system transitions from static fixed-capacity deployment to dynamic elastic scaling, allowing the computing resource quantity to fluctuate smoothly between high and low demand states while maintaining service availability.
4Quantity of substance
If virtual nodes are prematurely terminated to reduce costs, then resource consumption is reduced, but task execution reliability deteriorates
Solution Approach 1:
The scheduling system uses feedback from hardware utilization rate monitoring to make informed decisions about virtual node termination. By monitoring actual utilization patterns, the system can distinguish between truly idle virtual nodes that can be safely terminated and those that are actively processing tasks or approaching load thresholds, thereby maintaining task execution reliability while reducing resource consumption.
Solution Approach 2:
The system performs preliminary monitoring and assessment of virtual node utilization states before initiating termination actions. By evaluating hardware utilization rates and task execution status in advance, the system can identify appropriate termination candidates that will not disrupt ongoing tasks, ensuring reliable task completion while achieving cost reduction through premature termination of genuinely idle nodes.
Data Source
AI summary
Embodiments of the present specification provide a task processing method including: determining current state information of initial virtual nodes based on a target task received, where the current state information is determined based on a physical computing unit corresponding to the initial virtual nodes; based on task type information of the target task, determining candidate virtual nodes corresponding to the task type information from the initial virtual nodes; and determining a corresponding target virtual node for the target task based on the current state information of the candidate virtual nodes, and executing the target task through the target virtual node, thereby meeting a need to accurately determine the target virtual node for the target task.