Video Semantic Segmentation Pipeline for Low-Power Real-Time ISP
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image signal processors in smartphones struggle to replicate human visual fidelity due to constraints in camera sensor size and quality, and existing segmentation models are resource-intensive, unsuitable for resource-scarce devices.
Innovation Solution
A video semantic segmentation pipeline that generates per-frame per-pixel semantic segmentation maps for real-time enhancement, using a neural network coupled with an IIR filter to improve temporal consistency and reduce power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If heavy neural network-based segmentation models are used, then segmentation accuracy is improved, but power consumption and runtime increase making them unsuitable for resource-scarce devices
Solution Approach 1:
The segmentation pipeline is divided into multiple stages: a lightweight semantic segmentation network for initial processing, an IIR filter stage for temporal smoothing, and a per-pixel enhancement stage. This segmentation of processing stages allows the system to achieve accurate segmentation while controlling power consumption by applying heavy processing only where necessary.
Solution Approach 2:
The patent changes the parameters of the segmentation model by using a lightweight network architecture instead of heavy neural networks, and by introducing temporal smoothing parameters through the IIR filter. This allows maintaining segmentation accuracy while reducing power consumption and runtime requirements.
2Manufacturing precision
If per-pixel enhancement is applied in real-time during preview, then video quality mimicking human vision is improved, but processing complexity and power consumption increase
Solution Approach 1:
The system performs preliminary semantic segmentation using a lightweight network during the preview phase, and then applies per-pixel enhancement only to regions identified as needing enhancement. This preliminary action reduces the overall processing complexity by avoiding unnecessary heavy processing throughout the entire video frame.
Solution Approach 2:
Instead of applying uniform enhancement to the entire video frame, the patent applies per-pixel enhancement locally only to regions identified by the segmentation map. This local quality approach maintains high video quality fidelity while reducing processing complexity and power consumption in regions where enhancement is not needed.
3Stability of the object's composition
If segmentation maps are generated for every frame, then temporal consistency is improved, but processing time and power consumption increase
Solution Approach 1:
The patent applies the IIR filter continuously across frames to maintain temporal consistency in the segmentation maps. The filter uses feedback from previous frames to smooth transitions, ensuring continuity of useful action while reducing the computational burden compared to generating completely new segmentation maps for every frame.
Solution Approach 2:
The IIR filter uses feedback from previous frame segmentation maps to generate corrected feature maps for current frames. This feedback mechanism maintains temporal consistency by leveraging historical information, reducing the processing time required for each individual frame while ensuring stable segmentation results.
Data Source
AI summary
A method and device are provided in which a video stream is captured by a user equipment (UE). A semantic segmentation network in a processor of the UE generates a first feature map based on a first frame the video stream. The first feature map includes first information for generating a first segmentation and confidence map for the first frame. The processor generates a second feature map for a second frame of the video stream based on the first feature map. The second feature map includes second information for generating a second segmentation and confidence map for the second frame. The processor generates the second segmentation and confidence map based on the second information.