Elastic Inference Service Dynamic Accelerator Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning inference workloads are costly and inefficient to run in production, often requiring overprovisioning of GPU capacity, which is wasteful and expensive, due to the difficulty in optimizing hardware resources for varying machine learning workloads.
Innovation Solution
An elastic machine learning/inference service that allows for dynamic attachment and detachment of accelerator slots, providing cost-efficient hardware acceleration by decoupling CPU, DRAM, and hardware acceleration resources, and supporting multiple precision modes, enabling efficient use of resources across multiple inferencing applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If GPU capacity is overprovisioned to handle varying machine learning workloads, then service availability is improved, but resource waste and cost increase
Solution Approach 1:
The system dynamically provisions and deprovisions accelerator slots based on real-time workload demands. The control plane monitors inference workload characteristics and automatically adjusts the number of active accelerator slots, transitioning from static overprovisioning to dynamic adaptive provisioning that matches supply with actual demand.
Solution Approach 2:
The system changes the operational parameters of accelerator slots by supporting multiple precision modes (FP32, FP16, INT8). This allows the same hardware resource to adapt its computational characteristics to match different workload requirements, improving efficiency without requiring additional hardware for each precision level.
2Productivity
If dedicated GPU resources are allocated to each inferencing application, then performance is improved, but device complexity and cost increase
Solution Approach 1:
The system creates a universal accelerator slot interface that can serve multiple different inferencing applications and workloads. A single accelerator slot can be dynamically allocated to different applications based on demand, with the control plane managing the multiplexing. This eliminates the need for dedicated hardware per application while maintaining performance through proper resource isolation and management.
Solution Approach 2:
The control plane acts as an intermediary layer between the compute plane (applications) and the accelerator slots (hardware). It manages the complex task of allocating, deallocating, and migrating accelerator slots between different applications, abstracting away the hardware complexity from individual applications while enabling efficient resource sharing.
3Adaptability or versatility
If hardware acceleration resources are coupled with CPU and DRAM, then system simplicity is maintained, but adaptability to different precision modes and workloads decreases
Solution Approach 1:
The system segments the hardware acceleration resources into independent accelerator slots that can be individually managed, allocated, and configured. Each accelerator slot is a discrete unit that can be dynamically provisioned to different applications with different precision requirements, allowing the system to support multiple precision modes (FP32, FP16, INT8) without requiring monolithic hardware designs.
Solution Approach 2:
The control plane serves as an intermediary that manages the decoupling between CPU/DRAM resources and accelerator slots. It handles the complexity of resource allocation, ensuring that compute resources and acceleration resources are properly matched and coordinated, while allowing each component to be independently optimized and adapted to different workload requirements.
Data Source
AI summary
Implementations detailed herein include description of a computer-implemented method to migrate a machine learning model from one accelerator portion (such as a portion of a graphical processor unit (GPU)) to a different accelerator portion. In some instances, a state of the first accelerator portion is persisted, the second accelerator portion is configured, the first accelerator portion is then detached from a client application instance, and at least a portion of an inference request is performed using the loaded at least a portion of the machine learning model on the second accelerator portion that had been configured.