Model Platform Scheduling for Collaborative Rescheduling and Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of efficiently managing computing resources in model platforms to balance cost and performance, where deploying more resources leads to waste during low demand but insufficient resources cause service delays during high demand, necessitating a resource scheduling solution that improves utilization and reduces costs.
Innovation Solution
A model platform-based scheduling method that involves collaborative scheduling of models on a computing cluster in both the model orchestration and load balancing dimensions, using an optimization model to determine a target scheduling strategy that optimizes model deployment and traffic allocation, ensuring efficient resource utilization and reduced latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If more computing power is deployed to ensure efficient service performance, then service quality is improved, but service cost increases and computing resources are wasted during low demand periods
Solution Approach 1:
The patent implements dynamic resource scheduling that automatically adjusts computing resource allocation based on real-time inference demand. The system monitors traffic patterns and dynamically scales computing power deployment, transitioning from static over-provisioning to adaptive resource management that matches supply with actual demand conditions.
Solution Approach 2:
The system changes key operational parameters including resource allocation ratios, scheduling priorities, and load distribution coefficients based on demand conditions. By adjusting these parameters dynamically, the system optimizes the balance between service performance and resource utilization efficiency under different traffic scenarios.
2Loss of energy
If fewer computing resources are used to control cost, then service cost is reduced, but the model platform becomes prone to paralysis or severe service delays during inference demand surges
Solution Approach 1:
The system performs preliminary resource preparation and pre-positioning based on predicted demand patterns. By anticipating inference demand surges and pre-allocating computing resources in advance, the system ensures capacity availability when needed while avoiding continuous over-provisioning during low-demand periods.
Solution Approach 2:
The patent implements feedback mechanisms that continuously monitor service performance metrics, resource utilization rates, and demand patterns. This feedback loop enables the system to detect early signs of demand surges and automatically adjust resource allocation to maintain service stability while controlling costs.
3Productivity
If more computing nodes are deployed to handle inference traffic, then load capacity is improved, but resource utilization rate decreases due to instability of inference demands
Solution Approach 1:
The patent merges multiple scheduling dimensions including model orchestration and load balancing into a unified collaborative scheduling framework. This integration enables coordinated resource allocation across computing nodes, improving both load capacity and utilization efficiency by eliminating suboptimal decisions from isolated scheduling approaches.
Solution Approach 2:
The system creates a universal scheduling mechanism that handles multiple objectives simultaneously - load distribution, resource utilization optimization, and service performance maintenance. This multi-functional scheduling approach allows the same infrastructure to efficiently serve varying demand patterns without requiring dedicated resources for each function.
Data Source
AI summary
The present disclosure relates to a model platform-based scheduling method, medium and device. The method includes: receiving a model rescheduling request for a model platform; determining, from the computing cluster, a plurality of target computing nodes participating in rescheduling according to the model rescheduling request; determining a target scheduling strategy that satisfies a collaborative scheduling objective of the collaborative scheduling in the model orchestration dimension and the load balancing dimension, according to node information of the plurality of target computing nodes, model information corresponding to models carried by the plurality of target computing nodes and inference traffic information carried by the plurality of target computing nodes, combined with an optimization model; and rescheduling the models carried by the target computing nodes according to the target model orchestration mode, and controlling allocation of inference traffic to the target computing nodes according to the target inference traffic allocation mode.