Dynamic Inference Routing Between CPU and GPU
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning applications are computationally intensive, and existing methods have primarily focused on accelerating model training rather than optimizing inference processing, leading to increased workloads despite the use of specialized hardware like GPUs.
Innovation Solution
A computer-implemented method and system that dynamically determine a selected computing platform (CPU or GPU) for processing inference requests based on optimized objectives such as throughput, latency, power budget, and server node count, using profile information of the model and real-time status feedback of the computing platforms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If machine learning workloads are processed using specialized hardware like GPUs, then computational speed is improved, but device complexity and resource management complexity increase
Solution Approach 1:
The system dynamically selects between CPU and GPU based on real-time workload characteristics and system state, rather than using a fixed assignment. The routing decision changes adaptively based on model profile information, current queue depths, and performance metrics, resolving the contradiction by making the system flexible rather than static
Solution Approach 2:
The system changes operational parameters by switching between different computing platforms (CPU/GPU) based on varying workload parameters. Profile information about models (size, complexity, expected throughput) is used to determine optimal parameter settings, allowing the system to adapt to different computational requirements
2Productivity
If more computing resources are allocated to handle increased machine learning workloads, then productivity is improved, but energy consumption increases
Solution Approach 1:
The system adjusts energy consumption parameters by selecting appropriate computing platforms based on workload characteristics. For less demanding tasks, the system can route to lower-power CPU options, while reserving high-power GPU resources for computationally intensive workloads that require maximum throughput
Solution Approach 2:
The system monitors its own state (queue depths, current utilization) and automatically makes routing decisions without external intervention. This self-regulating mechanism ensures resources are used efficiently based on actual demand rather than constant high allocation
3Adaptability or versatility
If dynamic routing decisions are made based on real-time status feedback, then system adaptability is improved, but processing latency increases due to additional decision-making steps
Solution Approach 1:
Model profile information (size, complexity, expected throughput) is pre-computed and stored before runtime. When a workload arrives, the system can immediately match it against pre-analyzed profiles and make routing decisions without performing complex real-time analysis, thus maintaining adaptability while minimizing decision latency
Solution Approach 2:
The system uses real-time status feedback (current queue depths, utilization metrics) to inform routing decisions. This feedback mechanism allows the system to adapt to changing conditions while using simple, fast comparison logic rather than complex optimization algorithms
Data Source
AI summary
A method for dynamically assigning an inference request is disclosed. A method for dynamically assigning an inference request may include determining at least one model to process an inference request on a plurality of computing platforms, the plurality of computing platforms including at least one Central Processing Unit (CPU) and at least one Graphics Processing Unit (GPU), obtaining, with at least one processor, profile information of the at least one model, the profile information including measured characteristics of the at least one model, dynamically determining a selected computing platform from between the at least one CPU and the at least one GPU for responding to the inference request based on an optimized objective associated with a status of the computing platform and the profile information, and routing, with at least one processor, the inference request to the selected computing platform. A system and computer program product are also disclosed.