Variable Rate Shading Masks for Low-Compute Video ROI Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current ROI detection techniques in cloud gaming require significant computational resources and are sensitive to training data, leading to inefficiencies in encoding quality and bandwidth usage.
Innovation Solution
A novel ROI detection method based on variable rate shading (VRS) masks, which shares metadata between a renderer and encoder to optimize shading rates, reducing computation and improving coding efficiency by allocating more bits to detail-rich areas.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional ROI detection techniques (HOG Detector, DPM, CNN detectors) are applied to every frame in a video sequence, then encoding quality in regions of interest is improved, but computational resources and processing time are significantly consumed
Solution Approach 1:
The system performs preliminary action by generating a VRS mask during the rendering stage before encoding. This mask pre-identifies regions requiring high shading rates (and thus likely containing important visual information) without requiring separate ROI detection processing during encoding. The renderer outputs both the video frame and an associated VRS mask that guides subsequent encoding decisions.
Solution Approach 2:
The VRS mask serves multiple functions: it guides shading rate allocation during rendering and simultaneously serves as the basis for ROI detection during encoding. This multi-functionality eliminates the need for separate ROI detection algorithms (such as HOG or CNN detectors), reducing computational complexity while maintaining encoding quality in important regions.
2Manufacturing precision
If traditional ROI detection techniques are applied to every frame, then encoding quality is improved, but bandwidth usage increases due to processing overhead
Solution Approach 1:
The system extracts only the essential information needed for ROI detection from the rendering process itself (the VRS mask), rather than applying comprehensive traditional ROI detection algorithms to every frame. This extraction approach retrieves region importance information with minimal processing overhead, reducing the energy and bandwidth consumption associated with encoding while maintaining quality in important regions.
3Productivity
If VRS masks are used for ROI detection, then computation requirements are reduced and coding efficiency is improved, but encoding quality may be compromised in complex scenes
Solution Approach 1:
The system incorporates feedback by using the VRS mask generated during rendering to guide encoding decisions. The encoder receives both the video frame and the VRS mask, and uses the mask's shading rate information to dynamically adjust encoding parameters (such as quantization parameter) in different regions. This feedback loop ensures that regions requiring high visual quality receive appropriate encoding resources while maintaining overall coding efficiency.
Data Source
AI summary
Methods, apparatus, systems, and articles of manufacture to identify regions of interest of an image based on a variable rate shading (VRS) mask created from a video image. A method to detect the regions can include detecting, with one or more processors, regions of interest of the video image based on the VRS mask, the VRS mask generated by a renderer. The method can also include encoding, with one or more processors, the regions of interest of the video image, the encoding to include adjusting a quantization parameter suitable to encoding the regions of interest.