Executable Graph Optimization for Higher AI Inference Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI model inference technologies face inefficiencies in utilizing the processing capabilities of devices like CPUs, GPUs, TPUs, and FPGAs due to suboptimal executable graph and primitive kernel implementations, leading to lower-than-expected performance and resource underutilization.
Innovation Solution
An executable graph optimizer duplicates and optimizes multiple graphs to improve instruction and register utilization, reduces cache miss rates, and adjusts affinity modes, generating an optimized executable graph without modifying the primitive kernels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a trained model is converted into an executable graph and run on primitive implementation kernels, then AI inference can be performed, but device resource utilization is suboptimal and performance is lower than expected
Solution Approach 1:
The patent merges multiple same nodes from multiple executable graphs into a single optimized node. By duplicating the executable graph M times (where M ranges from 2 to N) and then merging corresponding nodes across these duplicated graphs, the system achieves better resource utilization and improved inference throughput while reducing redundant computations
Solution Approach 2:
The patent optimizes node parameters by analyzing instruction utilization and register utilization metrics. It adjusts node shapes, data types, and computational parameters to maximize device resource utilization. The system also modifies affinity modes and cache management parameters to reduce cache miss rates and improve overall performance
2Productivity
If the executable graph is optimized by merging nodes from multiple graphs, then instruction utilization and register utilization improve, but the complexity of the optimization process increases
Solution Approach 1:
The optimization process is segmented into distinct phases: graph duplication, node analysis, eligibility determination, and merging. Each phase handles specific tasks independently, making the complex optimization process more manageable and systematic
Solution Approach 2:
The system performs preliminary analysis of nodes to determine eligibility for merging before actually executing the merge operation. It pre-calculates instruction utilization and register utilization metrics, and identifies candidate nodes in advance, which streamlines the subsequent optimization process
3Productivity
If nodes are optimized by merging from multiple executable graphs, then cache miss rate reduces and performance improves, but memory requirements increase
Solution Approach 1:
The system creates M duplicated graphs where M is between 2 and N, which is an excessive action beyond the single original graph. This partial duplication enables better optimization opportunities while the merging process subsequently reduces the actual memory footprint, achieving a balance between optimization benefits and memory constraints
Data Source
AI summary
The application relates to optimization of an executable graph for AI model inference. An optimization method may include: duplicating the executable graph to generate a number M of same executable graphs; determining one or more nodes eligible for optimization from the executable graph, based on an inference throughput related parameter associated with an inference device to perform the AI model inference; and generating an optimized executable graph for the AI model inference by optimizing the one or more nodes from each of the number M of same executable graphs. Here, M is an integer in a range of 2 to a maximum number N of allowed executable graphs, and N is an integer manually configured or estimated based on a memory size of the inference device and a size of the executable graph.