XR Virtual Object Culling via 3D Occlusion Masking

Solution Overview

Problem

Existing imaging systems for generating extended-reality (XR) images struggle with accurate object occlusion between real and virtual objects, leading to improper culling and positioning, which reduces realism and immersiveness in XR environments.

Innovation Solution

An imaging system comprising a visible-light camera, pose-tracking means, and a processor that captures images, generates a three-dimensional model of the real-world environment, creates an occlusion mask, and culls virtual objects based on this mask to ensure accurate occlusion, with adjustments made based on predefined width percentages to prevent improper culling or positioning.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If existing imaging systems perform object occlusion between real and virtual objects, then virtual content can be integrated into XR environments, but improper culling and positioning of virtual objects occurs, reducing realism and immersiveness

Engineering Contradiction:

Improveintegration of virtual content into XR environmentsVSAvoidculling and positioning accuracy of virtual objects

Core Design Contradiction:

Adaptability or versatilityVSManufacturing precision

Solution Approach 1:

The system performs preliminary culling of virtual objects based on the occlusion mask before embedding them in the XR image. The processor determines which portions of virtual objects should be culled by comparing their positions with the occlusion mask representing real objects, ensuring accurate occlusion is established before final image generation.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The occlusion mask serves as an intermediary element between real objects and virtual objects. It is generated from the three-dimensional model of the real-world environment and used to determine how virtual objects should be culled and positioned, mediating the interaction between real and virtual content to achieve realistic occlusion.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If virtual objects are culled to achieve occlusion by real objects, then occlusion can be implemented, but the culled virtual objects may be improperly positioned on top of or too far from the real objects

Engineering Contradiction:

Improveocclusion implementationVSAvoidpositioning accuracy of culled virtual objects

Core Design Contradiction:

ReliabilityVSMeasurement precision

Solution Approach 1:

The system uses feedback from the occlusion mask and the spatial relationship between real and virtual objects to adjust the positioning of culled virtual objects. The processor continuously monitors the position of virtual objects relative to real objects and makes adjustments to ensure they are properly positioned, preventing them from appearing on top of or too far from the real objects.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system transitions from two-dimensional image processing to three-dimensional spatial reasoning by using a three-dimensional model of the real-world environment to generate the occlusion mask. This allows the system to consider depth and spatial relationships in three dimensions when culling and positioning virtual objects, ensuring accurate placement in the XR environment.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

3Adaptability or versatility

If existing imaging systems generate XR images with virtual content, then immersive XR environments can be created, but the XR images appear unnatural to the user due to improper object occlusion

Engineering Contradiction:

Improvecreation of immersive XR environmentsVSAvoidrealism and natural appearance of XR images

Core Design Contradiction:

Adaptability or versatilityVSManufacturing precision

Solution Approach 1:

The system performs preliminary culling of virtual objects based on the occlusion mask before embedding them in the XR image. The processor determines which portions of virtual objects should be culled by comparing their positions with the occlusion mask representing real objects, ensuring accurate occlusion is established before final image generation.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The occlusion mask serves as an intermediary element between real objects and virtual objects. It is generated from the three-dimensional model of the real-world environment and used to determine how virtual objects should be culled and positioned, mediating the interaction between real and virtual content to achieve realistic occlusion.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11568552B2Imaging systems and methods incorporating improved culling of virtual objects

Publication Date: 2023.01.31 VARJO TECH OY

AI summary

An imaging system including visible-light camera(s), pose-tracking means, and processor(s). The processor(s) is/are configured to: control visible-light camera(s) to capture visible-light image, whilst processing pose-tracking data to determine pose of camera(s); obtain three-dimensional model of real-world environment; create occlusion mask, using three-dimensional model; cull part of virtual object(s) to generate culled virtual object(s), wherein virtual object(s) is to be embedded at given position in visible-light image; detect whether width of culled part or remaining part of virtual object(s) is less than predefined percentage of total width of virtual object(s); if width of culled part is less than predefined percentage, determine new position and embed entirety of virtual object(s) at new position to generate extended-reality image; and if width of remaining part is less than predefined percentage, cull entirety of virtual object(s).