Deep Learning Compiler Scheduling for Heterogeneous Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning compilers lack the capability to deploy neural network models efficiently on heterogeneous accelerator platforms, failing to optimize computational performance and inference speed due to insufficient consideration of accelerator combinations.
Innovation Solution
A deep learning compiling method that utilizes a deployment table to guide the compiler in splitting and deploying neural network models across multiple hardware backends, selecting optimal schedules and execution times to enhance inference speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a neural network model is compiled using a conventional deep learning compiler on a single hardware backend, then the compilation process is simple, but the inference speed and processing performance are limited
Solution Approach 1:
The patent segments the neural network model into multiple operation groups, where each group can be independently compiled and executed on different hardware backends. This segmentation allows the system to distribute computational tasks across multiple accelerators (GPU, NPU, FPGA, etc.), thereby improving inference speed without requiring the entire compilation process to manage all hardware complexities at once. The intermediate representation is divided into operation groups that can be selectively deployed.
Solution Approach 2:
The patent creates a universal intermediate representation that can be compiled for multiple hardware backends simultaneously. This intermediate representation serves as a common interface that can be transformed into various hardware-specific code, allowing the same neural network model to be deployed across different accelerator types (GPU, NPU, FPGA, ASIC) without requiring separate compilation pipelines for each hardware platform.
2Productivity
If multiple hardware backends are used to execute neural network operations, then processing performance improves, but the complexity of selecting and coordinating schedules across backends increases
Solution Approach 1:
The patent performs preliminary analysis and selection of hardware backends before the actual compilation process. The system pre-evaluates which hardware backends are available and suitable for specific operation groups, and pre-determines the compilation strategy. This preliminary action includes analyzing the neural network model's operation groups and matching them with appropriate hardware backends based on capabilities and performance characteristics, thereby reducing the complexity during the actual compilation and execution phases.
3Productivity
If the neural network model is optimized for a specific hardware backend, then execution efficiency on that backend improves, but the model's adaptability to different hardware platforms decreases
Solution Approach 1:
The patent applies local quality optimization by allowing different operation groups within the same neural network model to be optimized for different hardware backends. Instead of uniformly optimizing the entire model for a single backend, the system identifies specific operation groups that benefit from particular hardware characteristics (e.g., convolution operations on GPU, specific neural network layers on NPU) and applies targeted optimizations locally to each group, thereby maintaining high execution efficiency while preserving adaptability across platforms.
Data Source
AI summary
A deep learning compiling method for neural network model is provided. A number of predetermined execution time values required to execute a number of predetermined schedules for each operation of a neural network model in a number of different hardware backends are estimated. Based on the predetermined execution time values, one of the predetermined schedules corresponding to each operation of the hardware backends is selected as a candidate schedule corresponding to each operation of the hardware backends. The candidate schedule corresponding to each operation of the hardware backends and the corresponding candidate execution time value are recorded in a table. An intermediate representation of the neural network model is split according to the table. According to the split intermediate representations and the table, a deep learning compiling process is executed to perform the operations on at least two different hardware backends.