Elastic Inference Service Dynamic Accelerator Slot Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning inference workloads are costly and inefficient to run in production due to the challenge of balancing CPU and accelerator usage, leading to overprovisioning and wasteful expenses, as existing technologies struggle to optimize hardware resources effectively.
Innovation Solution
An elastic machine learning/inference service that allows for the dynamic attachment and detachment of accelerator slots to inference applications, providing cost-efficient hardware acceleration by decoupling CPU and memory resources, and supporting multiple precision levels for efficient computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If GPUs are used to improve performance and efficiency of deep learning inference workloads, then computational performance is improved, but infrastructure cost increases due to overprovisioning capacity
Solution Approach 1:
The patent segments the GPU into multiple accelerator slots that can be independently allocated to different inference workloads. Instead of provisioning entire GPUs, the system divides the accelerator into smaller allocatable units (slots) that can be dynamically assigned based on actual workload requirements, reducing overprovisioning while maintaining performance.
Solution Approach 2:
The patent implements dynamic allocation and deallocation of accelerator slots to inference workloads based on real-time demand. The system can dynamically adjust the number of active accelerator slots per workload, allowing efficient resource utilization across multiple tenants without requiring overprovisioned static capacity.
2Productivity
If accelerator resources are virtualized and dynamically allocated, then resource utilization efficiency is improved, but maintaining performance and reliability objectives becomes more difficult
Solution Approach 1:
The patent implements monitoring and management of accelerator slot allocations, with the ability to track performance metrics and reliability status. The system can use feedback from workload performance data to adjust allocations, ensuring that dynamic resource sharing does not compromise reliability or performance guarantees for individual inference workloads.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Implementations detailed herein include description of a computer-implemented method. In an implementation, the method at least includes receiving an application instance configuration, an application of the application instance to utilize a portion of an attached accelerator during execution of a machine learning model and the application instance configuration including: an indication of the central processing unit (CPU) capability to be used, an arithmetic precision of the machine learning model to be used, an indication of the accelerator capability to be used, a storage location of the application, and an indication of an amount of random access memory to use.