GPU Training Job Packing for Higher Utilization and Lower Energy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning training processes face inefficiencies in utilizing graphics processing unit (GPU) resources due to resource-intensive data sets, leading to suboptimal utilization and increased costs and energy consumption.
Innovation Solution
A method and system for adaptively packing training jobs across multiple GPUs based on an efficiency cost model, utilizing a GPU gatekeeper to partition resources and determine optimal packing patterns, enabling efficient distribution and utilization without sacrificing accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If training jobs are distributed across multiple GPUs without adaptive packing, then GPU resources can be utilized, but resource utilization efficiency is suboptimal and costs increase
Solution Approach 1:
The system dynamically determines packing patterns for training jobs based on real-time GPU resource availability and job characteristics. The gatekeeper continuously adjusts the packing configuration rather than using a static allocation scheme, allowing the system to adapt to changing workloads and maximize GPU utilization efficiency while minimizing energy consumption.
Solution Approach 2:
The invention changes the parameter of job packing configuration by determining optimal packing patterns that fit multiple training jobs into available GPU memory spaces. By varying the packing density and arrangement of jobs across GPUs, the system improves resource utilization without proportionally increasing energy consumption.
2Productivity
If more GPU resources are allocated to training jobs, then training can be performed, but costs and energy consumption increase
Solution Approach 1:
The gatekeeper merges multiple training jobs into packed configurations that share GPU resources. By combining several smaller training jobs into a single packed workload that utilizes one or more GPUs, the system achieves better resource utilization and reduces the total number of GPUs needed, thereby lowering both costs and energy consumption while maintaining training throughput.
3Productivity
If training jobs are packed into fewer GPU resources, then resource utilization improves, but packing complexity increases
Solution Approach 1:
The gatekeeper autonomously determines the optimal packing patterns for training jobs without requiring manual intervention or complex external orchestration. The system self-manages the allocation and packing decisions by evaluating job characteristics and GPU availability, simplifying the overall system architecture while achieving high GPU utilization.
4Productivity
If adaptive packing is implemented, then GPU utilization is maximized, but system complexity increases
Solution Approach 1:
The gatekeeper acts as an intermediary layer between training job submissions and GPU execution. It absorbs the complexity of adaptive packing decisions by intercepting job requests, determining optimal packing patterns, and managing GPU allocation. This intermediary approach maximizes resource utilization efficiency while shielding the rest of the system from the inherent complexity of dynamic resource management.
Data Source
AI summary
A computer-implemented method for training a machine learning model includes receiving a first training job at a processing device of a computer system having a plurality of graphics processing unit (GPU) resources, the first training job being part of a set of training jobs, and determining an amount of available memory in each GPU resource of the plurality of GPU resources. The method also includes loading the training job into one or more GPU resources with at least one second training job. The loading includes determining a cost model indicating an efficiency cost of each of a plurality of packing patterns, and packing the first training job and the second training job into the one or more GPU resources according to a packing pattern associated with a lowest efficiency cost.