3D Object Identification and Segmentation With VST Cameras

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for identifying and segmenting real-world objects in virtual environments are inaccurate and inefficient, leading to issues with realism and effectiveness in mixed reality experiences due to manual processes, 2D image segmentation limitations, and challenges in differentiating objects of varying sizes and distances.

Innovation Solution

A computer-implemented method using video-see-through cameras to capture images, determine camera poses, identify image segments, generate 2D masks, and project them onto 3D models to determine object characteristics, enabling precise delineation and interaction with real-world objects.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If manual processes are used to identify and segment objects, then users can control the selection process, but the process becomes time-consuming and error-prone

Engineering Contradiction:

Improveuser control over object selectionVSAvoidspeed of object identification

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The system automatically performs object identification and segmentation using computer vision algorithms, eliminating the need for manual user input. The system self-services by autonomously detecting objects, generating 3D models, and segmenting them without requiring user guidance or intervention.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual mechanical operations (pointing, outlining) with automated digital processing. Computer vision algorithms and machine learning models automatically analyze images and generate 3D representations, substituting the mechanical act of manual selection with automated computational processes.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Device complexity

If 2D image segmentation is used, then processing is simpler, but accuracy in differentiating objects of different sizes and distances is compromised

Engineering Contradiction:

Improvesimplicity of segmentation processVSAvoidaccuracy of object differentiation

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The system transitions from 2D image segmentation to 3D spatial segmentation by generating three-dimensional models of objects. This dimensional upgrade enables the system to differentiate objects based on their spatial properties (size, distance, shape) rather than just their 2D pixel appearance, significantly improving measurement precision.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Solution Approach 2:

The patent segments the 3D environment into distinct object regions by generating separate 3D models for each detected object. This segmentation allows the system to process and analyze each object independently in three-dimensional space, improving accuracy while maintaining manageable complexity through structured processing pipelines.

Inventive Principle:

Principle #1Segmentation

3Device complexity

If everything in the room is fused into a single mesh, then processing is simplified, but detecting individual objects becomes challenging

Engineering Contradiction:

Improvesimplicity of environment representationVSAvoiddifficulty of individual object detection

Core Design Contradiction:

Device complexityVSDifficulty of detecting and measuring

Solution Approach 1:

The system divides the unified 3D environment into separate object instances by generating individual 3D models for each detected object. This segmentation strategy maintains the simplicity of processing while enabling accurate individual object detection and analysis.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces image segmentation masks and 3D bounding boxes as intermediary structures that bridge the gap between the unified environment representation and individual object detection. These intermediaries enable the system to identify and separate objects within the overall 3D scene without compromising processing simplicity.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20250336219A1Three-dimensional object identification and segmentation

Publication Date: 2025.10.30 VARJO TECH OY

AI summary

A computer-implemented method including: capturing at least one image using at least one video-see-through camera of a display apparatus; determining a pose of the at least one VST camera from which the at least one image is captured; identifying image segments in the at least one image that represent different real-world objects in a real-world environment; generating a set of two-dimensional (2D) image masks corresponding to the image segments representing the different real-world objects, wherein a given 2D image mask corresponds to a given real-world object; and digitally projecting the 2D image masks of the set onto a three-dimensional (3D) model of the real-world environment, from a perspective of the pose of the at least one VST camera, to determine at least one of: 3D shapes, 3D locations in the real-world environment, 3D orientations, 3D sizes, of the different real-world objects.