AI Inference Graph Partitioning Across CPU, GPU, and VPU
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic devices with multiple processors face inefficiencies when executing complex AI models, leading to processor overload, underutilization, and synchronization issues, resulting in prolonged processing times.
Innovation Solution
A runtime hyper-heterogeneous optimization method using the H2O engine to analyze and partition AI models into computational graphs executable by multiple processing circuits such as CPU, GPU, and VPU, optimizing workload distribution and graph partition sizes based on bandwidth and performance prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If the CPU executes the whole AI model alone, then the model execution is simple to manage, but the CPU becomes overloaded and processing time increases
Solution Approach 1:
The patent divides the AI model into multiple computational graphs and partitions them across different processing circuits (CPU, GPU, VPU). Each processing circuit executes specific sub-graphs based on its capabilities, thereby distributing the computational load and improving overall processing speed while maintaining manageable complexity through systematic partitioning.
Solution Approach 2:
The patent introduces a new dimension of parallel execution by utilizing multiple processing circuits simultaneously. Instead of sequential execution on a single CPU, the system executes multiple computational graphs in parallel across different hardware units, significantly improving processing throughput.
2Device complexity
If one processor executes most model operations, then the system is simple to control, but other processors remain idle and synchronization overhead increases
Solution Approach 1:
The patent dynamically assigns computational graphs to different processing circuits based on real-time bandwidth availability and processing capabilities. The system monitors the state of each processor and adjusts the partitioning and assignment of computational tasks dynamically, ensuring that all processors remain actively utilized rather than idle, thereby reducing wasted time while maintaining controllable system complexity through adaptive scheduling.
3Productivity
If computational graphs are partitioned across multiple processing circuits, then processor utilization improves, but synchronization overhead and system complexity increase
Solution Approach 1:
The patent implements a feedback mechanism where the system monitors bandwidth availability, processing performance, and synchronization status of multiple processing circuits. Based on this feedback, the system dynamically adjusts the partitioning strategy and task assignment to optimize processor utilization while managing synchronization overhead. The feedback loop enables the system to adapt to changing conditions and maintain efficient operation despite the increased complexity of multi-processor coordination.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention provides an electronic device including a plurality of processing circuits is disclosed, wherein the apparatus includes a circuitry configured to perform the steps of: receiving a model an input data for execution; analyzing the model to obtain a graph partition size of the model; partitioning the model into a plurality of graphs based on the graph partition size, wherein each of the graphs comprises a portion of operations of the model; deploying the plurality of graphs to at least two of the processing circuits, respectively; and generating output data according to results of the at least two of the processing circuits executing the plurality of graphs.