Hybrid Tiled Video Streaming ROI Enhancement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for enhancing regions of interest (ROI) in video streams are inefficient, particularly in high-density panorama or immersive-type videos, as they rely on a-priori knowledge of ROIs, leading to bandwidth waste and computational inefficiencies, and require parallel decoding processes that are resource-intensive and dependent on scalable video codecs.
Innovation Solution
A method that uses a hybrid tiled streaming scheme, where media data is encoded into base and enhancement tile streams, allowing for independent decoding and blending of enhancement tiles with base video frames to create enhanced ROIs, leveraging recent video coding standards like HEVC for efficient decoding and resource management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If ROI-based video coding with SVC is used to enhance ROIs, then video quality in ROI areas is improved, but bandwidth is wasted due to transmitting enhancement layers for multiple possible ROIs that may not be relevant to the actual user interest
Solution Approach 1:
The video stream is divided into base layer and enhancement layer, where the base layer contains the full video at lower quality and the enhancement layer contains only the ROI area at higher quality. This segmentation allows selective transmission of enhancement data only for actually interested ROIs, avoiding bandwidth waste from pre-transmitting multiple possible enhancement regions.
Solution Approach 2:
Different quality levels are applied to different regions of the video frame. The ROI area receives enhanced quality through the enhancement layer while non-ROI areas remain at base layer quality. This local quality enhancement ensures bandwidth is spent only on improving visually important areas rather than uniformly enhancing the entire frame.
2Adaptability or versatility
If multiple enhancement layers are produced based on user profiles to cover different ROIs, then adaptability to different user interests is improved, but computational efficiency deteriorates as it becomes inefficient to produce and decode hundreds of enhancement layers
Solution Approach 1:
The system dynamically determines which ROI enhancement is needed based on actual user input (such as gaze detection or user selection) rather than pre-defining multiple static enhancement layers for all possible user profiles. This dynamic approach allows the system to adapt to different user interests on-demand while avoiding the computational burden of pre-producing and maintaining hundreds of enhancement layers.
3Reliability
If parallel decoding processes are used to decode base layer and enhancement layer independently, then decoding reliability is improved, but device complexity increases due to resource-intensive parallel processing requirements
Solution Approach 1:
The encoded video stream is segmented into base layer and enhancement layer with clear structural separation. The base layer can be decoded independently to provide a complete low-quality video, while the enhancement layer can be selectively decoded and combined with the base layer output. This segmentation enables flexible decoding strategies where the decoder can choose to process only the base layer or both layers based on available resources, maintaining reliability while reducing complexity requirements.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for enhancing a region of interest in video frames of a video stream is described. The method includes receiving media data defining base video frames of at least one base stream, said base stream being associated with one or more enhancement tile streams, an enhancement tile stream comprising media data defining tiled video frames comprising a tile, said tile comprising media data for enhancing visual content in a subregion of the image region of the base video frames; requesting media data of one or more enhancement tile streams, preferably one or more HEVC enhancement tile streams, the one or more enhancement tile streams being associated with media data for enhancing visual content in the at least one region of interest; generating base video frames on the basis of media data of the at least one base stream and generating tiled video frames on the basis of media data of the one or more enhancement tile streams, the tiled video frames comprising one or more tiles forming visual content the at least one region of interest; and, replacing or blending at least part of the visual content of said at least one region of interest in the video frames of said base stream with at least part of said enhanced visual content of the tiled video frames.