GPU Pod Controller for Multi-Service Resource Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data centers using Kubernetes for machine learning and deep learning, GPUs are underutilized as a single Kubernetes Pod cannot fully utilize the resources of graphic memories, registers, and threads, leading to a need for a technique to dispatch multiple Pods to the same GPU for higher utilization.
Innovation Solution
The method employs Kubernetes' customized resource and annotation functions, along with CUDA hijack, to register and dispatch virtual services, allocating GPU graphic memory and time slices based on calculating requests, ensuring efficient resource allocation across multiple services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If one Kubernetes Pod is assigned to a single GPU, then the Pod can have dedicated GPU resources, but the GPU utilization ratio is low because a single Pod cannot fully utilize the abundant GPU resources
Solution Approach 1:
The patent segments the GPU resources by introducing GPU Pods as intermediate management units. Each GPU Pod manages a portion of GPU resources (graphic memory, registers, threads) and can be shared by multiple Kubernetes Pods. This segmentation allows fine-grained resource allocation while maintaining dedicated resource guarantees, resolving the contradiction between reliability and productivity.
2Productivity
If multiple Kubernetes Pods are dispatched to share the same GPU, then the GPU utilization ratio increases, but resource allocation management becomes complex
Solution Approach 1:
The patent introduces GPU Pods as intermediary entities between Kubernetes Pods and physical GPUs. The GPU Pod controller manages resource allocation, monitoring GPU usage and dynamically allocating resources to multiple Kubernetes Pods. This intermediary layer abstracts the complexity of multi-Pod resource management, enabling high GPU utilization while keeping the management system organized and controllable.
Solution Approach 2:
The system implements dynamic resource allocation where GPU Pods can flexibly allocate graphic memory, registers, and threads to different Kubernetes Pods based on real-time calculating requests and current resource utilization rates. This dynamic adjustment capability allows the system to optimize GPU utilization while managing multiple Pods, reducing the complexity burden through adaptive resource management.
3Productivity
If GPU resources are allocated based on calculating requests, then resource allocation efficiency improves, but the system complexity increases due to need for monitoring and controlling GPU time slices and graphic memory
Solution Approach 1:
The patent implements a self-service resource allocation mechanism where the GPU Pod controller automatically monitors GPU usage metrics (graphic memory consumption, time slice utilization) and makes allocation decisions based on predefined thresholds and current resource states. The system self-adjusts resource distribution without requiring complex external control, improving allocation efficiency while managing system complexity through automated decision-making algorithms.
Data Source
AI summary
A method and system for sharing a same GPU by a plurality of services, a device and a storage medium are provided. The method includes: in response to receiving a request of creating GPU services, creating the corresponding GPU services according to the request, creating GPU Pods of a corresponding quantity according to the GPU services, and associating the GPU services with the GPU Pods (S1); creating Kubernetes Pods according to a configuration of the GPU Pods, associating the Kubernetes Pods with the GPU Pods (S2); in response to receiving a calculating request, according to the calculating request, determining a specification of a GPU graphic memory or GPU time slice required to be applied for, and comparing with a threshold specified by the GPU services (S3); in response to the specification of the GPU graphic memory or time slice being less than the threshold, reading current residual resource amounts of the GPU Pods and the Kubernetes Pods, and comparing with the specification of the GPU graphic memory or time slice (S4); and in response to the specification of the GPU graphic memory or time slice being less than a sum of the current residual resource amounts of the GPU Pods and the Kubernetes Pods, according to a current resource utilization rate, dispatching the GPU Pods and the Kubernetes Pods for calculation (S5).