Serverless Function Instance Scaling for GPU Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing use of artificial intelligence (AI) workloads as serverless functions in cloud computing systems leads to challenges in GPU utilization, as GPUs are not as flexible as CPUs in handling concurrent workloads effectively.
Innovation Solution
A computer-implemented method for managing instances of serverless functions in a cloud computing system, which involves obtaining service level objectives, command queue lengths for GPUs, and request queue lengths to calculate the number of instances to deploy, identify suitable compute nodes, and create instances on those nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If more GPU resources are added to handle increasing AI workloads as serverless functions, then the system can process more concurrent requests, but GPU utilization efficiency deteriorates due to GPUs not being as flexible as CPUs in handling concurrent workloads
Solution Approach 1:
The patent implements dynamic instance management that automatically adjusts the number of serverless function instances based on real-time workload conditions. The system monitors workload demand and dynamically scales instance deployment, ensuring that GPU resources are actively utilized only when needed, thereby maintaining high GPU utilization efficiency while handling varying concurrent request loads.
Solution Approach 2:
The system changes operational parameters by adjusting the number of function instances deployed based on workload metrics. By monitoring parameters such as queue lengths and active request counts, the system dynamically modifies instance deployment parameters to optimize GPU utilization, preventing both underutilization and overprovisioning of GPU resources.
2Reliability
If the number of serverless function instances is increased to meet service level objectives, then concurrent request handling improves, but resource allocation efficiency deteriorates due to potential over-provisioning or under-utilization
Solution Approach 1:
The patent implements a feedback-driven instance management system that continuously monitors workload conditions and service level objective compliance. The system uses this feedback to dynamically adjust instance deployment, ensuring that service level objectives are met while optimizing resource allocation. The feedback loop prevents both over-provisioning (wasting resources) and under-provisioning (failing to meet objectives).
Solution Approach 2:
The system performs preliminary actions by pre-calculating optimal instance deployment based on predicted workload patterns and service level requirements. This allows the system to proactively allocate resources before demand spikes occur, ensuring service level objectives are met while avoiding reactive over-provisioning.
Data Source
AI summary
Computer-implemented methods for managing instances of serverless functions in a cloud computing system are provided. Aspects include obtaining a service level objective for a serverless function, obtaining a command queue length for a graphical processing unit disposed on each of a plurality of compute nodes in the cloud computing system, and obtaining a request queue length of the serverless function. Aspects also include calculating a number of instances of the serverless function to deploy in the cloud computing system, wherein the number of instances is determined based on the service level objective and the request queue length of the serverless function, identifying compute nodes from the plurality of compute nodes to deploy each of the number of instances of the serverless function, and creating an instance of the serverless function on each of the identified compute nodes.