Semantic Image Segmentation With Multi-Exit Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current semantic segmentation models are computation- and memory-intensive, failing to meet real-time requirements on resource-constrained devices like smartphones, leading to high latency and impacting user experience and safety in applications such as autonomous vehicles.
Innovation Solution
A two-staged training process for Multi-Exit Semantic Segmentation (MESS) networks with parametrised early exits, optimizing the placement and architecture of segmentation heads, and employing a novel Regularised End-to-End training algorithm and Positive Filtering Distillation to enhance accuracy and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning models with numerous layers and dilated convolutions are used to maintain high-resolution feature maps and large receptive field, then segmentation accuracy is improved, but inference latency increases significantly
Solution Approach 1:
The network is divided into multiple stages with intermediate exits, where each stage can independently perform segmentation. This allows the system to exit early at shallower stages for simple images, reducing latency, while still having access to deeper stages for complex images requiring higher accuracy.
Solution Approach 2:
The network dynamically selects which exit to use based on input image characteristics and device capabilities. The system can adaptively choose between early exits (faster, lower accuracy) and later exits (slower, higher accuracy), making the inference process dynamic rather than static.
2Measurement precision
If computation-intensive deep learning models are deployed on resource-constrained devices, then segmentation quality is improved, but device resource consumption increases
Solution Approach 1:
The system performs only the necessary computation required for each specific input. For easy images, computation is stopped at early exits with fewer operations. For difficult images, computation continues to deeper exits. This partial action approach avoids performing excessive computations on all inputs.
Solution Approach 2:
The effective network depth and complexity are changed as a parameter based on input difficulty and device resources. The system can switch between different computational configurations (different exits) to optimize the trade-off between quality and resource consumption.
3Productivity
If early exits are added to reduce inference latency, then processing speed is improved, but model complexity increases
Solution Approach 1:
Multiple exits share the same backbone feature extraction network, making the system multi-functional. The single backbone serves all exits, and each exit can handle different types of inputs. This universality reduces overall complexity compared to having separate networks for each processing level.
Solution Approach 2:
The feature extraction backbone is merged and shared across multiple exits rather than having separate extraction networks. This combining of functions reduces redundant computation and simplifies the overall model structure while still providing multiple processing paths.
Data Source
Figure 1~2
Figure 3(A)~3(C)
Figure 4~5(B)
AI summary
Broadly speaking, the present techniques generally relate to a method for training a machine learning, ML, model to perform semantic image segmentation, and to a computer-implemented method and apparatus for performing semantic image segmentation using a trained machine learning, ML, model. The training method enables a semantic image segmentation ML model that is able to make predictions faster, without significant loss in accuracy. The training method also enables the ML model to be implemented on apparatus with different hardware specifications, i.e. different computational power and memory, for example.