Concurrent Inference Scheduling Across Heterogeneous Computing Resources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Heterogeneous computing platforms face challenges in intelligently allocating computing resources to maximize resource utilization while efficiently executing inference tasks.
Innovation Solution
A method for scheduling concurrent inference tasks that determines the actual execution time and resource switching time for each model unit, allowing for rational allocation of computing resources to maximize resource utilization and ensure rapid task completion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If heterogeneous computing platforms process concurrent inference tasks, then inference efficiency is improved, but resource allocation optimization becomes insufficient
Solution Approach 1:
The patent segments the inference task into multiple model units that can be independently scheduled and executed on different computing resources. Each model unit is assigned to specific computing resources based on its characteristics, allowing fine-grained resource allocation and improving both inference efficiency and resource utilization without excessive complexity.
Solution Approach 2:
The patent implements dynamic resource allocation by continuously monitoring execution time and switching time of model units, and adjusting the scheduling strategy in real-time. The system dynamically selects computing resources based on current workload and performance metrics, enabling adaptive optimization of resource allocation while maintaining high inference efficiency.
2Quantity of substance
If computing resources are allocated to multiple inference tasks, then resource utilization rate increases, but execution time management becomes complex
Solution Approach 1:
The patent performs preliminary calculation of execution time and switching time for each model unit before actual task execution. By pre-evaluating the time costs of different resource allocation scenarios, the system can select optimal scheduling strategies in advance, avoiding complex real-time decision-making and reducing execution time overhead while maximizing resource utilization.
Solution Approach 2:
The patent implements feedback mechanisms by monitoring actual execution time and switching time of model units, and using this information to continuously optimize resource allocation decisions. The system adjusts scheduling parameters based on observed performance, enabling efficient management of multiple inference tasks while maintaining high resource utilization rates.
Data Source
AI summary
Provided is a method for scheduling concurrent inference tasks, an electronic device and a storage medium, relating to the fields of artificial intelligence, deep learning, large model and other technologies. The method includes: determining multiple types of computing resources and a plurality of network models required for the concurrent inference tasks, wherein the concurrent inference tasks represent a plurality of inference tasks to be processed in parallel, and each network model is used to execute at least one of the plurality of inference tasks; determining actual execution time required for each model unit in the plurality of network models to execute a task on a candidate computing resource as well as actual resource switching time corresponding to each model unit, to obtain total execution time required to execute the concurrent inference tasks; and using the total execution time to determine a target scheduling result.