Edge Inference Scheduler for Heterogeneous Computing Units
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning frameworks struggle to efficiently manage inference processes in heterogeneous edge environments due to the need for manual code writing for different hardware architectures and lack of custom scheduling schemes, leading to unsatisfactory performance and resource inefficiency.
Innovation Solution
An open and dynamic scheduler architecture that configures edge devices based on computational graphs to dynamically adjust processing methods, utilizing parallel algorithms like data, model, and pipeline parallelism, and generates target code for optimal execution across various hardware devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual code writing is used for different hardware architectures, then custom scheduling schemes can be implemented, but device complexity and development time increase significantly
Solution Approach 1:
The patent implements a universal scheduling framework that can handle multiple hardware architectures (CPU, GPU, FPGA, etc.) and different parallel algorithms through a single unified codebase. The scheduler automatically adapts to various computing units without requiring manual code writing for each architecture, thus achieving multi-functionality while reducing complexity.
Solution Approach 2:
The patent introduces a computational graph as an intermediary representation between the high-level inference description and the low-level hardware execution. This computational graph enables automatic translation and scheduling across different hardware architectures without requiring manual intervention, serving as a mediator that resolves the complexity issue.
2Productivity
If dedicated computing resources are used in cloud environments, then machine learning model training efficiency improves, but resource utilization in edge environments becomes inefficient
Solution Approach 1:
The patent implements dynamic resource allocation and scheduling that adapts to the specific capabilities and availability of edge devices. The system can dynamically adjust the execution plan based on the computational graph analysis and the actual state of edge computing resources, enabling efficient utilization of heterogeneous edge devices rather than static dedicated cloud resources.
Solution Approach 2:
The patent segments the machine learning inference process into discrete functions represented in a computational graph. This segmentation allows independent scheduling and allocation of different computation tasks to appropriate edge devices, enabling fine-grained resource utilization across the edge computing network rather than treating inference as an monolithic task.
3Productivity
If computing units are selected based on dependency relationships, then inference performance improves, but scheduling complexity increases
Solution Approach 1:
The patent performs preliminary analysis of the computational graph to identify dependency relationships between functions before execution. By pre-processing the computational graph to determine execution order and dependencies, the system prepares the scheduling information in advance, which simplifies the actual execution phase and improves inference performance without adding runtime complexity.
Data Source
AI summary
Implementations of the present disclosure relate to a method, an electronic device, and a computer program product for managing an inference process. Here, the inference process is implemented based on a machine learning model. A method includes: determining, based on a computational graph defining the machine learning model, dependency relationships between a set of functions for implementing the inference process; acquiring, in at least one edge device located in an edge computing network, a set of computing units available to execute the inference process; selecting at least one computing unit for executing the set of functions from the set of computing units; and causing the at least one computing unit to execute the set of functions based on the dependency relationships. With example implementations of the present disclosure, the inference process is implemented by making use of a variety of computing units in the edge computing network, thereby improving performance.