Graph Partitioning for Batch-Level Parallelism in Deep Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning frameworks face challenges in utilizing batch-level parallelism across multiple hardware units, requiring manual programming efforts to split data and assign concurrent inference execution instances, which is inefficient and complex, especially when dealing with additional inputs/outputs or unsupported operators.
Innovation Solution
An automated system that partitions deep learning models into clusters that either support batching or do not, allowing for autonomous scheduling of asynchronous inference executions on multiple hardware units, eliminating the need for manual intervention and optimizing performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual programming is used to split data and assign concurrent inference execution instances to each hardware unit, then batch-level parallelism can be achieved, but the system complexity and programming effort increase significantly
Solution Approach 1:
The system performs self-service by automatically analyzing the computational graph, identifying batch-compatible operations, and partitioning the graph into clusters that can be executed in parallel. The framework autonomously schedules inference executions across multiple hardware units without requiring developer intervention, making the system self-configuring and self-optimizing for batch-level parallelism
Solution Approach 2:
The computational graph is segmented into multiple batches based on operation compatibility with batch processing. The system divides the graph into batch-compatible clusters and non-batch clusters, allowing selective parallel execution of batched operations across multiple hardware units while maintaining proper data flow and dependencies
2Productivity
If automated graph partitioning is implemented to exploit batch-level parallelism, then execution efficiency improves, but the system architecture becomes more complex
Solution Approach 1:
The system performs preliminary analysis of the computational graph during model loading or compilation phase, identifying which operations support batch processing and how they can be clustered. This advance preparation creates an execution plan that enables efficient parallelism during inference without adding complexity during actual execution
Solution Approach 2:
An intermediary layer is introduced between the model definition and hardware execution that handles the complexity of graph analysis, batch identification, and clustering. This intermediary automatically translates the computational graph into optimized batched execution plans, shielding developers from architectural complexity while delivering performance benefits
3Productivity
If multiple hardware units are utilized for batched inputs, then compute throughput increases, but coordinating execution across units becomes more difficult
Solution Approach 1:
The system dynamically adapts the execution plan based on the specific computational graph structure and available hardware resources. It automatically adjusts which operations are batched, how they are clustered, and how they are distributed across hardware units, making the coordination transparent and adaptive to different scenarios
Data Source
AI summary
Systems, apparatuses and methods provide technology for batch-level parallelism, including partitioning a graph into a plurality of clusters comprising batched clusters that support batched data and non-batched clusters that fail to support batched data, establishing an execution queue for execution of the plurality of clusters based on cluster dependencies, and scheduling inference execution of the plurality of clusters in the execution queue based on batch size. The technology can include identifying nodes of the graph as batched or non-batched, generating a batched cluster comprising a plurality of batched nodes based on a relationship between two or more of the batched nodes, and generating a non-batched cluster comprising a plurality of non-batched nodes based on a relationship between two or more of the non-batched nodes. The technology can also include generating a set of cluster dependencies, where the cluster dependencies are used to determine an execution order for the clusters.