ML Device Placement via Graph Clustering and Reinforcement Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing device placement optimization techniques for machine learning models are inefficient, particularly for GPUs, due to factors like data movement, operation availability, and relationships between operations, leading to suboptimal performance in real-time production environments.
Innovation Solution
A computer-implemented method that uses graph data to cluster operations into CPU-only and GPU-available clusters, processes these clusters using a second machine learning model to determine optimal device placement, and adjusts model parameters using reinforcement learning based on performance metrics like throughput and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If GPU is used for machine learning model processing, then compute performance is improved, but data movement time increases significantly
Solution Approach 1:
The patent segments the computational graph into multiple independent subgraphs that can be processed in parallel. By dividing the model into separable components, data can be processed locally on GPU without requiring constant data movement between CPU and GPU, thus maintaining high compute performance while reducing data movement overhead.
Solution Approach 2:
The patent performs preliminary analysis of the computational graph to identify operations that can be executed on GPU versus CPU before actual inference. This pre-planning allows data to be positioned on the appropriate device in advance, minimizing data movement during actual model processing and maximizing GPU utilization.
2Productivity
If device placement optimization is applied, then model throughput is improved, but system complexity increases
Solution Approach 1:
The patent implements self-service through automated device placement optimization that analyzes the computational graph and autonomously determines optimal CPU-GPU distribution without requiring manual intervention. The system self-adjusts placement strategies based on graph characteristics, achieving high throughput while keeping the user interface simple.
Solution Approach 2:
The patent changes key parameters such as batch size, parallelization degree, and device allocation based on the analyzed computational graph characteristics. By dynamically adjusting these parameters rather than fixing them, the system achieves optimized throughput across different model types without requiring complex manual configuration.
3Productivity
If operations are distributed across CPU and GPU, then processing efficiency is improved, but operation availability constraints increase
Solution Approach 1:
The patent implements dynamic device placement that adapts to different operation types and availability constraints. Rather than statically assigning operations to fixed devices, the system dynamically determines optimal placement based on operation characteristics, GPU availability, and dependency relationships, maintaining processing efficiency while accommodating diverse operation requirements.
Solution Approach 2:
The patent introduces an intermediary layer that manages the complexity of CPU-GPU operation distribution. This intermediary analyzes operation availability constraints and mediates between computational requirements and device capabilities, allowing efficient distributed processing while shielding users from the complexity of operation-specific device constraints.
Data Source
AI summary
Systems, methods, and computer program products that use unsupervised learning to learn relationships between operations of a machine learning model based on a model graph representation to group the operations into clusters and, given a set of clusters and labels for the clusters, use a reinforcement learning algorithm to generate a final device placement result for the machine learning model.