Distributed Inference Model Partitioning for Lower Communication Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed environments, executing inference models across multiple data processing systems is challenging due to insufficient computing resources and increased communication bandwidth usage, leading to computational overhead and resource inefficiency.
Innovation Solution
The system partitions the inference model into portions, represented as a bipartite graph, and distributes these portions across data processing systems based on available computing resources and communication bandwidth, reducing inter-system communications and conserving resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the inference model is executed on a single data processing system, then the computing resources are sufficient, but the communication bandwidth consumption increases and system throughput decreases
Solution Approach 1:
The inference model is divided into multiple portions that are distributed across different data processing systems. Each portion is executed on a separate system, allowing parallel processing and reducing the communication bandwidth required compared to executing the entire model on a single system.
Solution Approach 2:
The system transitions from a single-node execution model to a multi-node distributed execution model. By adding the dimension of distribution across multiple data processing systems, the system achieves higher throughput while managing communication bandwidth through strategic partitioning.
2Productivity
If the inference model is partitioned and distributed across multiple data processing systems, then the computing resources are utilized efficiently, but the computational overhead increases
Solution Approach 1:
The inference model is segmented into portions that can be distributed across multiple systems. This segmentation allows efficient utilization of available computing resources while the partitioning strategy is designed to minimize the computational overhead associated with coordination and communication between systems.
3Power
If additional data processing systems are added to execute the inference model, then the computing capacity increases, but the communication bandwidth requirements increase
Solution Approach 1:
The model is segmented into portions that are distributed across multiple systems. This segmentation enables the system to scale computing capacity by adding more data processing systems while the partitioning strategy minimizes the increase in communication bandwidth requirements by reducing inter-system communications.
Data Source
AI summary
Methods and systems for inference generation are disclosed. To manage inference generation, a system may include an inference model manager and any number of data processing systems. The inference model manager may represent an inference model as a bipartite graph in order to obtain portions of the inference model. Each portion of the inference model may be distributed to one data processing system so that the data processing systems may collectively generate inferences usable by a downstream consumer. Portions of the inference model may be obtained so that each portion matches the available computing resources of a data processing system throughout the distributed environment. In addition, the portions may be obtained in order to reduce inter-data processing system communications during execution of the inference model.