Compiler Graph Partitioning for ML Inference Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional compiler approaches for machine learning (ML) inference models are time-consuming, power-intensive, and laborious due to the complexity of analyzing and processing each individual layer of large, complex model graphs, leading to inefficient data and control flow management.
Innovation Solution
A compiler that partitions the original model graph into sub-graphs based on operation type and latency, with unsupported operators executed by a general processor and supported operators by ML hardware, optimizing data movement and resource usage through automated, intelligent graph partitioning and multi-level structured metadata generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a compiler analyzes and processes each individual operator (layer) of a model graph, then the data and control flow can be generated for each layer, but the process becomes time-consuming and power-intensive due to the complexity and large number of layers
Solution Approach 1:
The model graph is partitioned into sub-graphs based on operator types and latency characteristics. This segmentation allows the compiler to process smaller, more manageable units rather than analyzing every individual operator sequentially, thereby reducing compilation time while maintaining accurate data and control flow generation for each layer.
2Productivity
If a compiler processes each individual operator (layer) of a model graph, then execution optimization can be achieved, but the process becomes laborious and unpractical given the complexity and large number of layers
Solution Approach 1:
By dividing the complex model graph into smaller sub-graphs based on operator types and latency, the system maintains execution optimization capabilities while making the compilation process more manageable and automated, reducing the manual intervention and complexity associated with processing each individual operator.
Solution Approach 2:
The automated graph partitioning system enables the compiler to autonomously organize and optimize model graphs without requiring manual intervention. The system automatically identifies operator types, calculates latency, and partitions graphs into appropriate sub-graphs, making the compilation process more practical and easier to operate despite the complexity of large model graphs.
3Reliability
If manual partitioning of model graph into sub-graphs is performed, then control and data flow can be generated, but the process becomes laborious and unpractical given the complexity and large number of layers
Solution Approach 1:
The system implements automated graph partitioning that autonomously analyzes operator types, calculates latency characteristics, and divides the model graph into sub-graphs without manual intervention. This automation maintains accurate control and data flow generation while eliminating the laborious manual partitioning process that was previously required.
Solution Approach 2:
The manual mechanical process of graph partitioning is replaced with an automated computational system that uses algorithms to analyze operator characteristics and perform partitioning. This substitution maintains the reliability of control and data flow generation while making the process practical for large, complex model graphs that would be impossible to manually partition.
Data Source
AI summary
A system and method for splitting a machine learning (ML) graph is disclosed. The system includes a compiler configured to receive an ML model. The compiler generates a graph associated with the ML model, wherein the graph is an internal representation of the ML model. The graph is partitioned into a first subgraph and a second subgraph. The first subgraph is associated with an ML hardware, an ML emulator, or a combination thereof, and the second subgraph is associated with a processor different from the ML hardware. A set of low-level instructions associated with the first subgraph is generated. One or more resources in the ML hardware is identified to execute the set of low-level instructions associated with the first subgraph.