Neural Network Compilation via Convolution Mapping for Faster Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network models based on structures like Transformer suffer from low inference speed due to operations such as dot-product, matrix-multiplication, and linear operations, which are not efficiently supported by conventional neural network processors.
Innovation Solution
Convert these operations into convolution operations supported by neural network processors, generating a target instruction sequence to accelerate the neural network model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network models use operations such as dot-product, matrix-multiplication, and linear operations, then the model can perform complex computations, but the inference speed becomes slow because these operations are not efficiently supported by conventional neural network processors
Solution Approach 1:
The patent changes the operational parameters by converting dot-product, matrix-multiplication, and linear operations into convolution operations. This parameter transformation allows the neural network model to execute on convolution-optimized processors, significantly improving inference speed while maintaining computational equivalence through mathematical reformulation of the operations
Solution Approach 2:
The patent introduces an operation conversion mechanism that acts as an intermediary between the original neural network operations and the processor's native convolution operations. This intermediary layer translates unsupported operations into equivalent convolution operations, enabling efficient execution without requiring hardware modification or sacrificing computational accuracy
2Productivity
If conventional neural network processors are used, then the hardware architecture is simple and well-established, but they cannot efficiently execute operations like dot-product and matrix-multiplication that are essential for Transformer-based models
Solution Approach 1:
The patent makes convolution operations universal by demonstrating that they can replace multiple different operation types (dot-product, matrix-multiplication, linear operations). This multi-functionality approach allows a single convolution-optimized processor architecture to efficiently handle diverse computational tasks previously requiring specialized hardware support for each operation type
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Disclosed in the embodiments of the present disclosure are a compilation method and apparatus for a neural network model, an inference method and apparatus for a neural network model, and a device and a medium. The compilation method comprises: acquiring a first model corresponding to a neural network model to be subjected to compilation; using, as a target network layer, any one of at least one network layer which includes a preset operation in the first model, and converting the preset operation of the target network layer into a convolution operation which is supported by a neural network processor, so as to obtain a converted first network layer, wherein the preset operation is an operation comprising a multiply - accumulate operation; on the basis of the converted first network layer, obtaining an optimization model corresponding to the first model; and according to the optimization model, generating a target instruction sequence that is executable by the neural network processor and corresponds to said neural network model. In the embodiments of the present disclosure, a preset operation comprising a multiply-accumulate operation may be implemented by means of a convolution operation, such that the preset operation of a model is accelerated by means of a neural network processor, thereby facilitating an improvement in the inference speed of the model.