XR Image Signal Processing With Preliminary Occlusion Culling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image rendering technologies for extended-reality (XR) images are computationally intensive and time-consuming due to the processing of real-world object data that will not be visible in the final XR image, as they are occluded by virtual objects, using occlusion culling techniques like Potentially Visible Set (PVS).
Innovation Solution
A system and method that performs occlusion culling by identifying virtual objects that occlude real objects in an XR environment, sending a VR image and information about occluded regions to a display apparatus, allowing the processor to skip reading out pixel data from these regions and control image processing parameters, thus reducing computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If occlusion culling techniques like Potentially Visible Set (PVS) are used to determine visible virtual objects, then the accuracy of determining which virtual objects are visible is improved, but the computational complexity and processing time increase because image data of real objects is still read out and processed even when they will be occluded
Solution Approach 1:
The system performs preliminary occlusion culling analysis using a depth map and virtual object data before actual image capture. By determining which real objects will be occluded by virtual objects in advance, the system can skip reading out pixel data for those occluded regions, thus reducing computational complexity while maintaining accurate visibility determination
Solution Approach 2:
The invention extracts and processes only the necessary data for occlusion determination (depth map and virtual object information) separately from the full image data. By isolating the occlusion analysis step and using it to guide subsequent image capture, the system avoids processing unnecessary real object data that would be occluded, thereby reducing overall computational load
2Reliability
If image data of all real objects is read out and processed to ensure accurate XR image generation, then the quality and realism of the final XR image is improved, but the processing time and energy consumption increase significantly
Solution Approach 1:
The system performs preliminary determination of occluded regions using depth map analysis and virtual object data before actual image capture. This preliminary action enables the system to skip reading out pixel data for regions that will be occluded, significantly reducing processing time while maintaining XR image quality by ensuring that only necessary real object data is processed
Solution Approach 2:
Instead of processing all real object data, the system applies partial processing by selectively reading out only the pixel data corresponding to non-occluded regions. This partial action approach maintains sufficient image quality for XR applications while dramatically reducing the time and energy required compared to processing complete image data
3Measurement precision
If the processor processes all captured image data to generate XR images, then the completeness and accuracy of the XR image is improved, but the energy consumption and computational load increase
Solution Approach 1:
The system performs preliminary occlusion analysis using depth map and virtual object data to identify regions that will be occluded before actual image capture. This preliminary action enables energy-efficient processing by allowing the processor to skip reading out and processing pixel data for occluded regions, thus reducing energy consumption while maintaining complete and accurate XR image generation for visible regions
Solution Approach 2:
The invention extracts and processes only the essential data elements for XR image generation (virtual object data and depth map information) separately from complete image data. By using this extracted information to guide selective image capture and processing, the system achieves complete and accurate XR images with significantly reduced energy consumption compared to processing all captured image data
Data Source
AI summary
Disclosed is a system comprising data repository(ies) and server(s) configured to: access, from data repository(ies), a first three-dimensional (3D) model of a virtual environment and a second 3D model of a real-world environment; combine the first 3D model and the second 3D model to generate a combined 3D model of an extended-reality (XR) environment; perform occlusion culling using the combined 3D model, to identify a set of virtual objects that occlude real object(s) in the XR environment from a perspective of a viewpoint; and send, to display apparatus(es), a virtual-reality (VR) image representing the set of virtual objects and information indicative of portion(s) of a field of view that corresponds to the real object(s) that is being occluded by the set of virtual objects from the perspective of the viewpoint.