Elastic Inference Service for GPU Resource Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning inference workloads are costly and inefficient to run in production due to overprovisioning of GPUs, leading to wasteful and expensive infrastructure costs.
Innovation Solution
An elastic inference service that allows for the attachment and detachment of accelerator slots, providing cost-efficient hardware acceleration by decoupling CPU and memory resources, and supporting multiple precision levels, enabling flexible and efficient use of hardware resources across multiple applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If GPUs are used to improve deep learning inference performance, then computational efficiency is improved, but infrastructure cost increases due to overprovisioning
Solution Approach 1:
The patent segments accelerator resources into virtual accelerator slots that can be independently allocated and attached to different compute instances. This segmentation allows fine-grained resource allocation, enabling multiple applications to share physical GPU resources without requiring full GPU provisioning for each application, thereby reducing infrastructure costs while maintaining computational efficiency.
Solution Approach 2:
The patent implements dynamic attachment and detachment of virtual accelerator slots to compute instances based on workload demands. This dynamic allocation allows the system to optimize resource utilization in real-time, ensuring computational efficiency is maintained when needed while reducing infrastructure costs during low-utilization periods through elastic scaling.
2Speed
If full GPU acceleration is provisioned for each application, then performance is improved, but resource utilization efficiency deteriorates due to overprovisioning
Solution Approach 1:
The patent creates virtual accelerator slots that can be universally attached to multiple different compute instances and applications. A single physical GPU can serve multiple virtual slots that are dynamically allocated to different applications based on their performance needs, enabling one resource to perform multiple functions and improving overall resource utilization efficiency while maintaining required inference performance.
Solution Approach 2:
The patent creates virtual copies of accelerator resources through virtual accelerator slots. These virtual copies can be attached to multiple compute instances simultaneously, allowing applications to access accelerator performance without requiring dedicated physical GPUs. The virtualization layer manages the copying and sharing of actual GPU resources, improving resource utilization while maintaining performance.
3Device complexity
If CPU and memory resources are coupled with accelerator resources, then system simplicity is maintained, but flexibility deteriorates due to inability to independently scale resources
Solution Approach 1:
The patent segments the system into independent resource components: compute instances with CPU and memory, and separately managed virtual accelerator slots. This segmentation allows each resource type to be independently provisioned, scaled, and attached based on specific workload requirements, providing flexibility while maintaining manageable system complexity through clear separation of concerns.
Solution Approach 2:
The patent introduces a virtualization layer as an intermediary between physical accelerator resources and compute instances. This intermediary manages the attachment and detachment of virtual accelerator slots to compute instances, enabling flexible resource allocation while abstracting the complexity of resource management from both the hardware and application layers, thus maintaining system simplicity.
4Stability of the object's composition
If accelerator resources are statically allocated, then system stability is improved, but adaptability deteriorates due to inability to respond to changing workload demands
Solution Approach 1:
The patent implements dynamic attachment and detachment of virtual accelerator slots to compute instances based on real-time workload demands. This dynamic approach allows the system to maintain stability through consistent performance during normal operation while adapting to changing conditions by弹性ally allocating or releasing accelerator resources, thus resolving the contradiction between stability and adaptability.
Solution Approach 2:
The patent incorporates feedback mechanisms that monitor workload demands and automatically adjust the allocation of virtual accelerator slots accordingly. This feedback-driven approach ensures system stability by maintaining appropriate resource levels while enabling adaptability to changing workload conditions, as the system continuously responds to performance metrics and demand signals.
Data Source
AI summary
Implementations detailed herein include description of a computer-implemented method. In an implementation, the method at least includes receiving an application instance configuration, an application of the application instance to utilize a portion of an attached accelerator during execution of a machine learning model and the application instance configuration including an arithmetic precision of the machine learning model to be used in determining the portion of the accelerator to provision; provisioning the application instance and the portion of the accelerator attached to the application instance, wherein the application instance is implemented using a physical compute instance in a first location, wherein the portion of the accelerator is implemented using a physical accelerator in the second location; loading the machine learning model onto the portion of the accelerator; and performing inference using the loaded machine learning model of the application using the portion of the accelerator on the attached accelerator.