Convolutional Kernel Pre-loading for Inference Time Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks face high inference times and memory occupancy demands, which are not effectively balanced by existing solutions, particularly in resource-constrained devices.
Innovation Solution
A method that optimizes temporary working memory by pre-loading and reusing convolution kernels, reducing the number of kernel recovery and loading processes, allowing for efficient calculation of output data without increasing volatile or non-volatile memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If convolution kernels are loaded repeatedly during calculation, then calculation accuracy is maintained, but inference time increases significantly
Solution Approach 1:
The patent applies preliminary action by pre-loading convolution kernels into temporary working memory before the actual calculation process begins. The system identifies available memory size, pre-loads the maximum number of kernels that can be stored, and then performs calculations using these pre-loaded kernels. This eliminates repeated loading operations during inference, reducing inference time by up to 98% for 2D convolutions while using only existing memory resources without increasing overall memory occupancy.
2Productivity
If TVM code generation is used to avoid loop executions, then calculation speed improves, but non-volatile memory size increases significantly
Solution Approach 1:
The patent applies segmentation by dividing the convolution calculation process into distinct phases: kernel pre-loading phase, calculation phase using pre-loaded kernels, and memory management phase. By segmenting the computation and pre-loading kernels in batches that fit within available temporary memory, the system achieves high calculation speed without requiring large non-volatile memory allocations for pre-computed results, thus avoiding the memory overhead of TVM-style code generation.
3Productivity
If weight stationary or feature stationary techniques are applied, then calculation efficiency improves, but volatile memory requirements increase
Solution Approach 1:
The patent applies dynamics by making the memory allocation adaptive rather than static. The system dynamically identifies the available temporary working memory size and adjusts the number of kernels to pre-load accordingly. This dynamic approach allows the system to achieve calculation efficiency similar to weight stationary or feature stationary techniques while adapting volatile memory usage to actual available resources, preventing excessive memory requirements.
Data Source
AI summary
In an embodiment a method for managing a convolutional calculation carried out by a calculation unit adapted to calculate output data on output channels from convolution kernels applied to input data blocks on at least one input channel, wherein calculations on each input data block correspond respectively to an output datum on an output channel, and wherein the calculations with each convolution kernel correspond to the output data on each output channel respectively includes identifying a size of a memory location available in a temporary working memory of the calculation unit, pre-loading in the temporary working memory a maximum number of convolution kernels storable at the size of the memory; and controlling the calculation unit to calculate a set of output data calculable from pre-loaded convolution kernels.