Shared GPU Runtime and API Proxy for Serverless Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GPU runtime systems in serverless computing are bulky, causing storage redundancy, cold start delays, and inefficient data transmission, which hinder efficient allocation and management of GPU resources for inference applications.
Innovation Solution
A lightweight GPU-sharing method using GPU streams to enable concurrent execution of multiple functions within a single GPU runtime, intercepting and forwarding GPU API calls through a proxy process to manage and allocate resources efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a GPU runtime is packaged with every container or process for inference tasks, then the task can execute independently with complete hardware context and library support, but storage occupancy increases significantly (up to 1.5 GB per function with 95% taken by GPU runtime) and deployment density is limited
Solution Approach 1:
The patent merges multiple container-shared GPU runtimes into a single shared runtime instance. Instead of each container having its own GPU runtime package (causing 1.5 GB per function occupancy), the system creates one GPU runtime that multiple containers can share, dramatically reducing redundant storage occupancy while maintaining independent execution capability through container isolation.
Solution Approach 2:
The GPU runtime is transformed from a container-specific component to a universal shared resource. The single GPU runtime instance serves multiple containers simultaneously, providing universal access to hardware context and machine learning libraries across different inference functions, thereby improving deployment density without sacrificing reliability.
2Reliability
If a GPU runtime is initialized for each inference function, then the function can run independently, but cold start delay occurs (5 to 8 seconds) which is unacceptable for short runtime inference tasks (200 ms)
Solution Approach 1:
The GPU runtime is initialized in advance and kept ready before inference tasks are executed. The system pre-creates a shared GPU runtime instance with all necessary hardware context and library configurations, so when an inference function needs to run, it can immediately utilize the pre-initialized runtime without undergoing the 5-8 second cold start process, reducing latency to match the actual inference runtime of 200 ms.
Solution Approach 2:
The shared GPU runtime acts as an intermediary between multiple containers and the physical GPU hardware. Instead of each container directly initializing its own runtime (causing repeated cold starts), the shared runtime mediates access to GPU resources, maintaining a persistent connection to the hardware and enabling multiple containers to share the same initialized runtime instance, thereby eliminating redundant cold start delays.
3Reliability
If different GPU runtimes are used for different functions, then each function has isolated address space for security and stability, but data transmission between functions involves CPU memory and causes redundant data copies with heavy time costs
Solution Approach 1:
The patent merges the address spaces of multiple functions into a single shared address space within the shared GPU runtime. Instead of maintaining separate isolated address spaces for each function (which require CPU-mediated data transmission), the system creates a unified address space that allows direct memory access between functions, eliminating redundant data copies and reducing transmission time while maintaining security through container-level isolation rather than runtime-level isolation.
Data Source
AI summary
A GPU-sharing method and apparatus for serverless inference loads is provided, wherein the method involves intercepting and forwarding GPU API calls made by inference tasks to an API proxy process to manage and allocate GPU resources. With a CPU and multiple GPUs connected through a bus, the GPUs communicate with the CPU only through an API proxy for process management and resource allocation of the GPUs. The CPU intercepts all GPU APIs triggered by any function of a same inference application, forwards the intercepted GPU APIs to a same designated GPU runtime for execution, and directs the GPU APIs triggered by each function to a pre-designated stream pool for the same inference application, so that all the functions of the same inference application share the same GPU runtime. The present disclosure solves the problem related to bulkiness of GPU runtimes in serverless inference systems, thereby facilitating GPU resource usage.