Composite Video Stream Generation for VR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Client devices face challenges in decoding multiple video streams simultaneously, particularly in VR environments where displaying a background video and foreground video streams is computationally intensive, especially for lower-end devices with only one hardware decoder.
Innovation Solution
Generating a composite video stream by obtaining spatially segmented encoding of the background video, determining the placement position of the foreground video, identifying occluded segments, selectively decoding and compositing image data, and encoding the composite video stream to reduce computational complexity and bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If multiple video streams are decoded separately on client devices, then each video stream can be processed independently, but the computational complexity and bandwidth usage increase significantly
Solution Approach 1:
The patent combines multiple independently decodable video streams into a single composite video stream by merging the background video and foreground video stream(s) into one encoded output. This allows the client device to decode only one stream instead of multiple separate streams, reducing computational complexity while maintaining the ability to process different video content independently through the compositing process performed by the server
2Adaptability or versatility
If multiple video streams are transmitted to client devices, then each stream can be delivered independently, but the bandwidth requirements increase
Solution Approach 1:
The patent merges multiple video streams into a single composite video stream for transmission to client devices. The server performs the compositing operation before encoding, resulting in one consolidated stream that contains both background and foreground video content. This reduces bandwidth consumption compared to transmitting multiple separate streams while maintaining the versatility of delivering different video content through the composite stream structure
3Ease of operation
If the entire background video is decoded, then all segments are available for compositing, but the computational load increases unnecessarily for occluded segments
Solution Approach 1:
The patent extracts and processes only the necessary portions of the background video that are not occluded by foreground video streams. The server identifies which background segments are visible and need to be decoded, and which segments are completely occluded and can be skipped. This selective decoding approach reduces computational load and energy consumption while maintaining the ease of operation by only processing relevant background segments
Data Source
AI summary
A processor system and computer-implemented method may be provided for generating a composite video stream which may combine a background video and a foreground video stream into one stream. For that purpose, a spatially segmented encoding of the background video may be obtained, for example in the form of a tiled stream. The foreground video stream may be received, for example, from a(nother) client device. The foreground video stream may be a real-time stream, e.g., when being used in real-time communication. The image data of the foreground video stream may be inserted into the background video by decoding select segments of the background video, inserting the foreground image data into the decoded background image data of these segments, and by encoding the resulting composite image data to obtain composite segments which, together with the non-processed segments of the background video, form a spatially segmented encoding of a composite video.