Dynamic Inference Operator Selection for Variable Input Sizes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for inference acceleration in convolutional layers are unstable due to the inability to meet dynamic acceleration requirements of inference models with diverse input configurations.
Innovation Solution
Configure multiple size intervals for inference models in advance, and select appropriate acceleration operators based on the target input size during execution, ensuring adaptive matching of operators for dynamic input sizes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If kernels are solidified and applied throughout the execution phase based on initialization phase matching, then inference acceleration is achieved during execution, but the approach cannot meet dynamic acceleration requirements of inference models with multiple input configurations
Solution Approach 1:
The patent applies dynamics by transitioning from static kernel matching (solidified in initialization phase) to dynamic kernel selection (performed in each execution phase). The system now selects acceleration operators based on actual input sizes during execution, enabling the inference model to adaptively match operators to diverse input configurations, thus resolving the contradiction between acceleration performance and adaptability.
2Productivity
If kernels are matched during initialization phase and solidified for entire execution phase, then computational efficiency is improved, but inference acceleration performance becomes unstable for dynamic input sizes
Solution Approach 1:
The patent applies preliminary action by pre-configuring multiple acceleration operators during the initialization phase, but not solidifying a single match. Instead, the system prepares a set of candidate operators and selects from them during execution based on actual input sizes, thus maintaining both computational efficiency (through pre-prepared operators) and stability (through adaptive selection).
Data Source
Figure 1
Figure 2~3
AI summary
Embodiments of the present disclosure provide an inference acceleration method, a device, and a storage medium. Multiple size intervals are configured for an inference model in advance, and adapted acceleration operators are respectively selected for size interval in advance. Based on this, during execution of the inference model, a target input size corresponding to a target inference operation during a current round of inference is determined, a target size interval including the target input size is found from the multiple size intervals configured for the inference model, and an acceleration operator associated with the target size interval is used as a target acceleration operator suitable for the target inference operation during the current round of inference. Therefore, during execution of an inference model, an adapted acceleration operator can be determined only by finding an appropriate size interval for a target inference operation. In this way, when the inference model supports a dynamic input size, adaptive matching of acceleration operators can be efficiently and accurately implemented, thereby improving inference acceleration performance.