AI Inference Graph Partitioning to Cut CPU-Accelerator Overhead

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing deep learning compilers inefficiently utilize accelerators and CPUs due to significant pre-processing overhead when switching between them, leading to suboptimal inference performance.

Innovation Solution

Categorize operations into CPU, accelerator, and undetermined types, and divide the computational graph into sub-graphs to minimize pre-processing steps by converting undetermined operations based on estimated processing times, ensuring operations are processed by the same unit type to reduce overhead.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If the compiler selects the accelerator for all operations supported by the accelerator, then the processing speed is improved, but the pre-processing overhead increases due to frequent data format conversions between CPU and accelerator

Engineering Contradiction:

Improveprocessing speedVSAvoidpre-processing overhead

Core Design Contradiction:

SpeedVSLoss of time

Solution Approach 1:

The computational graph is divided into sub-graphs based on processing unit types (CPU or accelerator). This segmentation allows operations to be grouped by their processing unit designation, minimizing the number of transitions between different processing units and thereby reducing pre-processing overhead while maintaining high processing speed for accelerator operations.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The compiler dynamically determines the optimal processing unit designation for operations in the computational graph by analyzing data flow patterns and processing requirements. This dynamic approach allows the system to adaptively assign operations to CPU or accelerator based on what minimizes pre-processing conversions, rather than using a static rule-based assignment.

Inventive Principle:

Principle #15Dynamics

2Adaptability or versatility

If data is processed by one execution unit and then by the other type, then the versatility of processing is improved, but the pre-processing overhead increases due to required data format conversions

Engineering Contradiction:

Improveprocessing versatilityVSAvoidpre-processing complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

Different regions of the computational graph are assigned different processing unit designations (CPU or accelerator) based on local characteristics such as operation type, data format requirements, and processing performance needs. This local optimization minimizes the number of transitions between processing units while maintaining the versatility to handle diverse operation types appropriately.

Inventive Principle:

Principle #3Local quality

3Loss of time

If operations are designated for processing by the same unit type, then the pre-processing overhead is reduced, but the processing versatility may be limited

Engineering Contradiction:

Improvepre-processing overheadVSAvoidprocessing flexibility

Core Design Contradiction:

Loss of timeVSAdaptability or versatility

Solution Approach 1:

The system maintains processing versatility by allowing operations to be designated for either CPU or accelerator based on their specific requirements. The computational graph analysis identifies operations that can be efficiently executed on either unit type and assigns them to minimize pre-processing overhead while preserving the ability to handle diverse computation types through the sub-graph division strategy.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12572339B2Accelerate inference performance on artificial intelligence accelerators

Publication Date: 2026.03.10 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method for inference performance in an artificial intelligence model provides reduction of pre-processing overhead. The method includes receiving a plurality of operations associated with the artificial intelligence model. A computational graph for the artificial intelligence model is generated. Each of the operations is categorized into one of three categories including: accelerator designated operations, central processing unit (CPU) designated operations, and undetermined processing designated operations. An estimated processing time is determined for the operations. The operations are inserted into the computational graph. The computational graph is divided into sub-graphs. Edges of the sub-graphs where pre-processing steps will be performed is determined. A conversion is applied to the sub-graphs converting the undetermined processing designated operations, into one of the accelerator designated operations or the CPU designated operations, based on a condition that minimizes a number of the pre-processing steps in the sub-graph.