Emulating Motion Parallax in VR via 3D Object Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current 360-degree video solutions in Virtual Reality (VR) lack motion parallax, leading to a less immersive experience, especially in three Degree-of-Freedom (DOF) systems, and full six DOF VR systems require significant computation, rendering, and bandwidth, making them impractical for widespread use due to tethered head-mounted display (HMD) limitations.
Innovation Solution
The system and method involve emulating motion parallax by processing 3D videos to remove objects, tracking HMD position changes, and rendering 3D models at the HMD, allowing for selective application of motion parallax effects based on user interest and network capabilities, using a combination of server-side and client-side rendering to optimize performance and bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full six DOF VR is implemented to enable motion parallax, then motion parallax and realism are improved, but computation, rendering, and bandwidth requirements increase significantly
Solution Approach 1:
The system segments the VR content into two components: a 360-degree video background and separate 3D model objects. This segmentation allows selective processing where only the necessary 3D objects are extracted and re-rendered with motion parallax, rather than processing the entire VR scene at full six DOF complexity.
Solution Approach 2:
The system extracts specific 3D objects from the 360-degree video using computer vision and depth estimation algorithms. By taking out only the relevant objects that need motion parallax effects and representing them as separate 3D models, the system reduces the overall computational burden while maintaining realism for key elements.
2Reliability
If full six DOF VR is implemented to enable motion parallax, then motion parallax and realism are improved, but the HMD becomes tethered
Solution Approach 1:
The system applies motion parallax selectively to specific 3D objects rather than the entire VR environment. By providing partial motion parallax effects only where needed (on extracted 3D objects) rather than excessive full six DOF processing everywhere, the system achieves acceptable realism with reduced computational requirements, enabling wireless HMD operation.
3Reliability
If 3D objects are extracted and re-rendered with motion parallax, then motion parallax is improved, but bandwidth requirements increase
Solution Approach 1:
The system performs preliminary extraction of 3D objects and generates optimized 3D models in advance. By preparing these objects beforehand with appropriate level-of-detail variations and caching them on the client device, the system reduces the bandwidth required during runtime, as only essential data needs to be transmitted rather than full-resolution 3D data.
Solution Approach 2:
The system applies different quality levels to different parts of the VR scene. Extracted 3D objects that require motion parallax receive higher processing quality, while the background 360-video maintains its original quality. This local quality approach optimizes bandwidth usage by focusing computational resources only where motion parallax is needed rather than uniformly processing the entire scene.
4Measurement precision
If 3D reconstruction and depth estimation are performed, then motion parallax accuracy is improved, but computation requirements increase
Solution Approach 1:
The system uses the existing 360-degree video itself as an intermediary to guide the depth estimation process. By comparing the extracted 3D objects against the original video frames and using the video's inherent depth cues, the system achieves accurate depth estimation without requiring complex external 3D reconstruction of the entire scene, thus reducing computational requirements.
Data Source
AI summary
Motion parallax effects can be emulated for 3D video content. At a head mounted display (HMD), motion parallax may be emulated through: receiving a 3D video at the HMD; obtaining a model of an object in the 3D video; obtaining a processed 3D video in which the 3D video is processed to remove the object from the 3D video; tracking a change in position of the HMD by a sensor of the HMD; rendering the processed 3D video at the HMD; and rendering the model of the object at a position in the processed 3D video based on the tracked change in position of the HMD. Multilayer spherical video, indicating the depths of objects therein, which may be used for motion parallax emulation may also be generated.