Elastic Inference Service for GPU Cost Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning inference workloads are costly and inefficient to run in production due to overprovisioning of GPUs, leading to wasteful and expensive infrastructure costs.
Innovation Solution
An elastic inference service that allows for the attachment and detachment of accelerator slots, providing cost-efficient hardware acceleration by decoupling CPU and memory resources, and supporting multiple precision levels, enabling flexible and efficient use of hardware resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If GPUs are used to improve deep learning inference performance, then computational efficiency is improved, but infrastructure cost increases due to overprovisioning
Solution Approach 1:
The patent segments accelerator resources into virtual slots that can be independently allocated and attached to different compute instances. This segmentation allows fine-grained resource allocation, eliminating the need to overprovision entire GPUs and reducing infrastructure costs while maintaining computational efficiency through precise resource matching.
Solution Approach 2:
The patent implements dynamic attachment and detachment of accelerator slots to compute instances based on workload demands. This dynamic allocation allows the system to adapt resource provisioning in real-time, improving computational efficiency when needed while reducing infrastructure costs during low-demand periods by detaching unused accelerators.
2Reliability
If accelerator resources are allocated to meet peak demand, then service reliability is improved, but resource utilization efficiency deteriorates due to overprovisioning
Solution Approach 1:
The patent creates a universal accelerator slot interface that can be attached to different compute instances serving various workloads. This universality allows the same physical accelerator resources to serve multiple functions and different customers dynamically, ensuring service reliability for each while improving overall resource utilization efficiency through shared access.
Solution Approach 2:
The patent changes the allocation parameter from static GPU assignment to dynamic accelerator slot attachment. This parameter change enables the system to maintain service reliability by attaching accelerators when needed while improving resource utilization by detaching and reallocating them to other workloads, eliminating the waste associated with permanent overprovisioning.
3Device complexity
If CPU and memory resources are coupled in traditional architectures, then system simplicity is maintained, but resource allocation flexibility deteriorates
Solution Approach 1:
The patent segments the traditional coupled CPU-memory-accelerator architecture into independent virtualizable components. Compute instances, accelerator slots, and storage resources are separated and can be independently allocated and attached based on specific workload requirements, providing resource allocation flexibility while maintaining manageable system complexity through virtualization abstraction.
Solution Approach 2:
The patent introduces a virtualization layer as an intermediary between physical hardware resources and compute instances. This intermediary enables flexible resource allocation by managing the attachment and detachment of accelerator slots to different compute instances, providing adaptability while hiding the underlying complexity from users through standardized interfaces.
4Measurement precision
If precision is increased to maintain inference accuracy, then model accuracy is improved, but computational cost increases
Solution Approach 1:
The patent enables local quality control by allowing different compute instances to attach to accelerator slots configured for specific precision requirements (e.g., FP16, BF16, INT8). This allows each workload to use the appropriate precision level for its specific inference tasks, maintaining accuracy where needed while reducing computational cost for workloads that can tolerate lower precision, optimizing the trade-off between accuracy and cost.
Data Source
AI summary
Implementations detailed herein include description of a computer-implemented method. In an implementation, the method at least includes receiving an application instance configuration, an application of the application instance to utilize a portion of an attached accelerator during execution of a machine learning model and the application instance configuration including: an indication of the central processing unit (CPU) capability to be used, an arithmetic precision of the machine learning model to be used, an indication of the accelerator capability to be used, a storage location of the application, and an indication of an amount of random access memory to use.