Neural Network Compilation for Heterogeneous Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current CPU+FPGA/ASIC heterogeneous computing platforms face challenges in programmability, reducing development thresholds, improving programming efficiency, optimizing neural network structures, and minimizing data movement between CPU and FPGA/ASIC.
Innovation Solution
A programming model for a neural network-oriented heterogeneous computing platform that includes a compilation method and runtime support system, utilizing a Neural Network (NN) optimizing compiler, assembler, and linker to generate hybrid ELF executable files, and a development library for initialization and runtime control of neural network-specific processors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If a conventional CPU-based general-purpose computing platform is used, then programmability and ease of development are maintained, but computing performance and processing speed for neural network workloads are insufficient
Solution Approach 1:
The system segments the computing platform into heterogeneous components: CPU for control and coordination, and FPGA/ASIC for accelerated neural network computation. The compilation system is also segmented into multiple independent tools (annotation tool, graph optimization tool, code generation tool, bitstream generation tool) that work together but can be developed and optimized independently, allowing each segment to address specific requirements without compromising overall system programmability.
Solution Approach 2:
The patent introduces several intermediary components to bridge the gap between high-level programming and hardware acceleration. The annotation tool serves as an intermediary that adds metadata to high-level code without changing the source code structure. The graph optimization tool acts as an intermediary that transforms annotated computational graphs into optimized execution plans. These intermediaries enable programmers to write code in familiar high-level languages while the system automatically handles the complexity of hardware-specific optimizations.
2Speed
If FPGA/ASIC-based acceleration is introduced to improve computing performance, then processing speed increases, but data movement overhead between CPU and FPGA/ASIC increases
Solution Approach 1:
The system performs preliminary actions by pre-processing neural network models through annotation and graph optimization before code generation. The annotation tool adds execution metadata to operations in advance, and the graph optimization tool performs fusion, pruning, and scheduling optimizations beforehand. This preliminary processing reduces the need for repeated data movements during execution, as the optimized computation graphs are directly translated into efficient hardware code that minimizes memory access patterns.
Solution Approach 2:
The patent merges multiple optimization functions into a unified compilation pipeline. The annotation tool, graph optimization tool, code generation tool, and bitstream generation tool are combined into an integrated system that processes neural network models end-to-end. This merging allows optimizations to be compounded and coordinated, ensuring that data movement optimizations are consistent across different stages of compilation and execution, thereby reducing overall data transfer overhead between CPU and accelerator.
3Productivity
If heterogeneous CPU+FPGA/ASIC platform is used to accelerate neural network computing, then computing performance improves, but development complexity and compilation complexity increase
Solution Approach 1:
The patent creates a universal annotation format that can represent various neural network operations and their attributes in a platform-independent manner. The computational graph structure serves as a universal intermediate representation that can model different neural network architectures and operations. This universality allows the same high-level description to be compiled for different target platforms (CPU, FPGA, ASIC) without requiring platform-specific source code, thereby reducing development complexity while maintaining high computing performance.
Solution Approach 2:
The compilation system implements self-service through automated optimization and code generation. The graph optimization tool automatically performs fusion, pruning, and scheduling optimizations without requiring manual intervention. The code generation tool automatically translates optimized computational graphs into platform-specific code and bitstreams. This automation eliminates the need for developers to manually optimize neural network models for different hardware platforms, significantly reducing development complexity while maximizing computing performance.
4Productivity
If manual optimization of neural network structure and compilation is performed, then computing efficiency can be improved, but programming efficiency and development time decrease
Solution Approach 1:
The system implements feedback mechanisms at multiple levels. The annotation tool provides feedback by automatically adding execution metadata based on operation types and data characteristics. The graph optimization tool uses feedback from the annotated graph to identify optimization opportunities and apply appropriate transformations. The code generation tool receives feedback from the optimized graph structure to generate efficient platform-specific code. This automated feedback loop enables the system to achieve manual-level optimization quality without requiring manual intervention, preserving programming efficiency while maximizing computing efficiency.
Data Source
AI summary
The disclosure provides a compilation method and system for heterogeneous computing platform, and a runtime method and system for supporting program execution on the heterogeneous computing platform. Inputting a trained neural network model to a Neural Network (NN) optimizing compiler to generate an NN assembly file corresponding to the neural network; inputting the NN assembly file to an NN assembler to generate an NN binary file corresponding to the neural network; compilation and assembling a neural network application developed by users in a high-level language using a host compiler toolchain to generate a corresponding host assembly file and a host binary file in sequence; and linking the NN binary file and the host binary file using a host linker to generate a single hybrid linking executable file. The technical solution of the present disclosure has the advantages such as good computing performance, strong scalability, strong compatibility and high flexibility.