Free-Viewpoint Image Synthesis for Low-Bitrate VR Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual reality technologies are limited by high data rates and restricted freedom of movement, failing to provide an immersive experience with real-life captured video due to the need for multiple video streams and high bitrates, and lack the flexibility to allow users to freely change viewing positions and directions.
Innovation Solution
A system that synthesizes images based on a stored database of real-life captured images, using a viewing position parameter vector to select and transform images, allowing users to dynamically change their position and direction with reduced bandwidth requirements by transmitting only one image stream, and adjusting for latency at the client side.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple cameras with different viewing positions are used to allow dynamic viewing position changes, then the freedom of viewing position selection is improved, but the data rate and system complexity increase significantly
Solution Approach 1:
The system segments the scene into multiple depth layers (foreground, midground, background) and processes each layer separately. This allows selective transmission of depth information only for relevant layers, reducing overall data rate while maintaining freedom of viewing position changes.
Solution Approach 2:
The patent adds a depth dimension to traditional 2D video by incorporating depth maps and performing depth-based rendering. This enables free-viewpoint video where users can change viewing positions without requiring multiple parallel video streams, as the depth information allows synthetic view generation from a single captured video sequence.
2Adaptability or versatility
If multiple parallel video streams are distributed to provide viewing position changes, then the adaptability to user position is improved, but the device complexity and processing requirements increase
Solution Approach 1:
The system performs preliminary depth extraction and layer segmentation during video capture or pre-processing. Depth maps are generated in advance for each depth layer, so that during playback only lightweight depth-based rendering operations are needed, avoiding complex real-time processing of multiple video streams.
Solution Approach 2:
Instead of transmitting multiple actual video streams from different cameras, the system creates synthetic copies of the scene by rendering 2D video frames with associated depth maps. These synthetic views are generated computationally rather than captured physically, reducing the number of required camera systems and data transmission requirements.
3Manufacturing precision
If depth maps and multiple view-points are encoded and transmitted, then the quality of three-dimensional virtual reality experience is improved, but the bitrate and data transmission requirements increase
Solution Approach 1:
The system applies different quality levels to different depth layers. Foreground objects with higher visual importance receive higher quality depth mapping and processing, while background objects use lower quality approximations. This selective quality allocation maintains overall 3D experience quality while reducing total bitrate requirements.
Solution Approach 2:
The patent transmits only the essential depth information needed for free-viewpoint rendering rather than complete three-dimensional models. By using depth maps from a limited number of captured views and interpolating intermediate depths, the system achieves adequate 3D quality with partial depth information, reducing bitrate compared to transmitting full multi-view stereo data.
4Manufacturing precision
If free-viewpoint video with multiple view-points is implemented, then the virtual reality experience quality is improved, but the freedom of movement and viewing direction changes remains restricted
Solution Approach 1:
By incorporating depth maps and performing depth-based compositing, the system adds a third dimension to traditional multi-view video. This enables continuous viewing position changes along the depth axis, not just discrete switching between pre-captured camera positions, thereby increasing freedom of movement within the virtual environment.
Solution Approach 2:
The system dynamically adjusts rendering parameters including viewing position, viewing direction, and depth layer composition based on user input. By changing these parameters in real-time and re-rendering the composite image from the desired viewpoint, the system provides continuous freedom of movement rather than restricted discrete position switching.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An apparatus comprises a store (201) storing images corresponding to different positions and viewing directions for a scene, and associated position parameter vectors for the images where the vector for an image comprises data indicative of a viewing position and a viewing direction. A receiver (205) receives a viewing position parameter vector from a remote client (101). A selector (207) selects a set of images in response to a comparison of the viewing position parameter vector and the associated position parameter vectors. An image synthesizer (209) generates an image from the set of images. A data generator (215) generates a reference position parameter vector for the synthesized image indicating a viewing position and direction for the synthesized image. An image encoder (211) encodes the synthesized image and an output generator (213) generates an output image signal comprising the encoded synthesized image and the reference position parameter vector. The output image signal is transmitted to the remote client.