VR Viewpoint Transformation Occlusion Compensation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing VR technologies face issues with occlusion regions generated during viewpoint transformation, leading to spatial discomfort and VR sickness due to incomplete matching of camera and eye positions, which existing methods like Passthrough+ struggle to fully address.
Innovation Solution
An information processing device and method that acquires color and depth images from different viewpoints, separates the depth image into foreground and background, and generates an output color image at a virtual viewpoint, compensating for occlusion regions by combining past and current depth and color information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If viewpoint transformation is performed to match camera and eye positions, then spatial comfort is improved, but occlusion regions are generated when objects shield the background
Solution Approach 1:
The system performs preliminary action by continuously estimating geometry information of the environment in real-time before viewpoint transformation occurs. This allows the system to prepare depth and color information for potential occlusion regions in advance, so when viewpoint transformation is performed, the occlusion regions can be properly compensated using pre-estimated geometry data from previous frames.
Solution Approach 2:
The system introduces geometry information as an intermediary element that mediates between the camera viewpoint and the eye viewpoint. By continuously estimating environment geometry and using it to compensate for occlusion regions, the system creates a bridge that allows accurate viewpoint transformation while maintaining complete background information even when objects temporarily shield areas from the camera view.
2Productivity
If a coarse mesh of 70×70 is used for depth estimation to reduce processing load, then productivity is improved, but artifacts occur when objects move forward
Solution Approach 1:
The system applies segmentation by dividing the depth estimation process into two distinct stages: a coarse estimation phase using a 70×70 mesh to reduce processing load, and a refinement phase that processes only the occlusion regions identified from geometry information. This segmentation allows the system to maintain low overall processing load while achieving high precision where it matters most - in the occlusion regions that require accurate depth and color compensation.
Solution Approach 2:
The system implements local quality by applying different processing qualities to different regions of the image. The majority of the image uses the efficient coarse mesh estimation, while only the occlusion regions receive refined processing with higher precision depth and color information. This ensures that computational resources are concentrated where they are most needed - in the regions that will be visible after viewpoint transformation - while maintaining overall productivity.
3Productivity
If minimum geometry information is handled to reduce processing load, then productivity is improved, but occlusion regions cannot be completely covered when viewpoint changes
Solution Approach 1:
The system maintains continuity of useful action by continuously estimating geometry information of the environment in real-time across multiple frames. This continuous estimation ensures that even when using minimum geometry information processing, the system accumulates and maintains up-to-date depth and color information for all regions including potential occlusion areas. When viewpoint transformation occurs, this continuous geometry data ensures complete coverage of occlusion regions without requiring heavy processing.
Data Source
AI summary
Provided is an information processing device configured to acquire a color image at a first viewpoint and a depth image at a second viewpoint, and generate an output color image at a virtual viewpoint different from the first viewpoint on a basis of a result of separation processing of separating the depth image into a foreground depth image and a background depth image.