Dynamic ROI Cropping With Lightweight Detection and ISP Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video conferencing systems face challenges in achieving high-quality, low-power dynamic region-of-interest (ROI) cropping due to trade-offs between image quality and resource consumption, with current methods requiring substantial power and bandwidth for high-resolution processing or resulting in blurry images after upscaling.
Innovation Solution
Implementing a lightweight detection/segmentation pipeline on the system's neural processing unit (NPU), graphics processing unit (GPU), or central processing unit (CPU) for initial ROI cropping, followed by in-pipeline cropping and scaling within the ISP pipeline, focusing on the ROI without the need for post-processing upscaling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If high-resolution processing is performed to maintain image quality, then image quality is improved, but power consumption and resource usage increase
Solution Approach 1:
The image processing is segmented into two stages: first, a lightweight detection/segmentation pipeline identifies the ROI at lower resolution; second, only the identified ROI region is processed at high resolution by the ISP pipeline. This segmentation allows the system to maintain high image quality for the important region while avoiding the power consumption of processing the entire high-resolution image.
Solution Approach 2:
The system applies different processing qualities to different regions of the image. The ROI region receives high-resolution processing with full ISP pipeline operations to maintain image quality, while the rest of the image is processed at lower resolution or discarded. This local quality approach ensures power is spent only where visually important.
2Manufacturing precision
If full image processing is performed before cropping, then image quality is maintained, but resource efficiency decreases due to processing discarded pixels
Solution Approach 1:
The system performs preliminary ROI identification using the lightweight detection pipeline before the main ISP processing occurs. By determining the ROI boundaries in advance, the system can then configure the ISP pipeline to process only the relevant region at high resolution, avoiding the waste of processing pixels that will ultimately be cropped out.
Solution Approach 2:
The essential information (ROI coordinates and boundaries) is extracted from the full image by the lightweight detection pipeline before the main processing occurs. This extracted information is then used to guide the ISP pipeline to process only the necessary region, effectively taking out only the important parts for full processing.
3Manufacturing precision
If ROI cropping is performed after full processing, then image quality is maintained, but power consumption increases due to processing unnecessary regions
Solution Approach 1:
The system dynamically adjusts the processing pipeline based on the detected ROI. The ISP pipeline is configured in real-time to process only the identified region of interest at high resolution, while the rest of the image receives minimal or no processing. This dynamic adaptation allows the system to maintain image quality for the important region while minimizing energy loss on unnecessary processing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems, apparatus, articles of manufacture, and methods are disclosed for high quality and low power dynamic region of interest (ROI) cropping. An example apparatus disclosed herein provides a first image to image signal processor (ISP) circuitry, the ISP circuitry to implement an image processing pipeline to process the first image. The example apparatus also downscales the first image to generate a second image having lower resolution than the first image and identifies a region of interest (ROI) in the second image. The example apparatus further provides coordinates of the ROI to the ISP circuitry, the ISP circuitry to crop the first image based on the coordinates and to output a third image based on the cropped first image.