Batch DNN Inference With Stitching and Padding for Small Images
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face inefficiencies when processing individual images with small pixel datasets due to underutilization of specialized vector and matrix accelerators, high latency from off-device memory access, and significant processing overhead, which results in poor utilization of processor capability.
Innovation Solution
Implementing a modified DNN-based inference model with a stitching layer to combine multiple images into a batch, a batch reshape layer to insert padding, and an un-stitch layer to process the batch efficiently, optimizing spatial dimensions and minimizing processing cycles using techniques like DMA and bitmasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If individual images are processed by specialized vector and matrix accelerators, then processing speed for single images is improved, but processor utilization deteriorates due to underutilization when images have small pixel datasets
Solution Approach 1:
The patent combines multiple individual image processing operations into a single batch processing operation. By merging N individual images into one batch, the system achieves better utilization of the SIMD processor's vector and matrix processing width while maintaining efficient processing speed through parallel operations on all images simultaneously.
Solution Approach 2:
The patent segments the processing workflow into distinct phases: batch formation phase (stitching layer combining multiple images), batch processing phase (DNN inference on the combined batch), and result separation phase (un-stitch layer dividing results). This segmentation allows optimization at the batch level while preserving individual image processing capabilities.
2Quantity of substance
If kernels or weights of trained DNN models are stored in off-device memory, then device memory constraints are satisfied, but access latency increases causing under-utilization of the accelerator
Solution Approach 1:
The patent performs preliminary action by pre-loading all necessary kernels and weights into device memory before batch processing begins. This preliminary loading ensures that during the actual DNN inference operation, no time is lost accessing off-device memory, thereby eliminating access latency while satisfying device memory constraints through careful memory management.
Solution Approach 2:
The patent ensures continuous useful action by organizing the batch processing pipeline so that data flows continuously through the accelerator without interruptions for memory access. The batch processing approach allows the accelerator to maintain full utilization throughout the processing of all images in the batch, eliminating idle time caused by sequential memory access.
3Ease of operation
If initial processing is performed for each input image, then individual image requirements are met, but processing overhead accumulates resulting in poor utilization of the processor
Solution Approach 1:
The patent merges identical initial processing operations that would otherwise be performed separately for each image into a single operation performed once on the entire batch. By combining the batch formation, DNN inference, and result separation into one unified processing flow, the system eliminates redundant overhead while maintaining the ability to handle individual image requirements.
Data Source
AI summary
Technology is disclosed herein to execute an inference model by a processor which includes a reshape layer. In an implementation, the reshape layer of the inference model receives an output produced by a previous layer of the inference model and inserts padding into the output, then supplies the padded output as an input to a next layer of the inference model. In an implementation, the inference model includes a stitching layer at the beginning of the inference model and an un-stitch layer at the end of the model. The stitching layer of the inference model stitches together multiple input images into an image batch and supplies the image batch as an input to a subsequent layer. The un-stitch layer receives output from a penultimate layer of the inference model and unstitches the output to produce multiple output images corresponding to the multiple input images.