Mixed Reality Teleoperation Vision Using Depth-Based Stereo

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional teleoperation systems using camera-based stereoscopic vision suffer from errors in depth estimation and can cause dizziness and sickness in operators due to inaccuracies in distance perception, which is critical for precise tasks.

Innovation Solution

A method that involves obtaining motion tracking data from a teleoperation client, transmitting it to a teleoperation object equipped with an optical and depth sensing device, generating a stereoscopic image based on monoscopic images and depth information, and displaying it to the client, allowing for precise distance estimation and improved vision quality.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If a conventional camera-based stereoscopic vision system is used in teleoperation, then the operator can have a stereoscopic view of the remote environment, but the depth estimation accuracy deteriorates and causes operator discomfort

Engineering Contradiction:

Improvedepth estimation accuracyVSAvoidoperator discomfort

Core Design Contradiction:

Measurement precisionVSObject-affected harmful factors

Solution Approach 1:

The patent introduces a depth sensing device (such as a time-of-flight camera or structured light sensor) as an intermediary component that actively measures depth information. This mediator provides accurate depth data that compensates for the inaccuracies in conventional passive stereoscopic vision, thereby improving depth estimation accuracy and eliminating operator discomfort without requiring changes to the fundamental stereoscopic display approach

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces the passive optical-mechanical stereoscopic system (relying solely on binocular parallax from two camera lenses) with an active sensing system that uses depth sensing technology. This substitution transitions from a mechanical/optical approach with inherent limitations to an electronic sensing approach that can directly measure depth, achieving superior depth accuracy and operator comfort

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Measurement precision

If a VR HMD or MR HMD device is used to provide stereoscopic vision, then the vision quality and resolution are improved, but the system complexity increases

Engineering Contradiction:

Improvevision qualityVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent integrates multiple functions into the teleoperation system by combining the depth sensing device with the existing camera system and HMD display. The depth sensing device serves dual purposes: providing accurate depth information for stereoscopic vision and enabling enhanced teleoperation control, thereby improving vision quality without proportionally increasing system complexity

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent merges the depth sensing capability with the visual display system by integrating the depth information from the depth sensing device with the monoscopic images from the camera. This combination creates an enhanced stereoscopic vision system where the HMD displays both visual and depth information in a unified manner, improving vision quality while maintaining manageable system complexity through integrated architecture

Inventive Principle:

Principle #5Merging (Combining)

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

This approach provides a high-precision stereoscopic image that enhances distance estimation and reduces operator discomfort, enabling more accurate and reliable remote operation of systems like robots and vehicles.

Implementation Method 1

obtaining, while the motion of the second object is controlled, a monoscopic image by the optical sensing device, wherein the monoscopic image comprises an image of at least one third object and a background, and depth information of the at least one third object by the depth sensing device

Methodology Applied

Scientific EffectTime of flight: Time of Flight

Data Source

PatentEP4198691A1A method and system of teleoperation with a remote streoscopic vision-through mixed reality

Publication Date: 2023.06.21 DEUTSCHE TELEKOM AG

Figure 1
Figure 2

AI summary

A method for providing a stereoscopic image in a teleoperation system is provided, comprising: obtaining motion tracking data for a motion of a first object at a teleoperation client, transmitting the motion tracking data of the first object from the teleoperation client to a teleoperation object, controlling a motion of a second object using the motion tracking data of the first object at the teleoperation object, wherein the second object comprises an optical sensing device and a depth sensing device, obtaining, while the motion of the second object is controlled, a monoscopic image by the optical sensing device, wherein the monoscopic image comprises an image of at least one third object and a background, and depth information of the at least one third object by the depth sensing device at a teleoperation object, generating a stereoscopic image of the at least one third object based on the monoscopic image and the depth information, and displaying the generated stereoscopic image at the teleoperation client