Fused Visual-Inertial-Odometry and Object Tracker for AR Anchoring

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current Visual-Inertial-Odometry (VIO) methods are not suitable for automatically attaching virtual content to physical objects in Augmented Reality due to their high tracking rate and lack of relation to video stream content, while Object Tracking (OT) is slower and less robust, requiring frequent restarts when objects are not visible.

Innovation Solution

A fusion of VIO and OT to create a fused-tracker that processes video frames in both coordinate systems, estimating a transform from the Object Tracking coordinate system to the VIO coordinate system, allowing for the projection of model-associated content anchored to physical objects, using a generalized Hough transform to determine the most probable transform and account for drift and outliers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If Visual-Inertial-Odometry (VIO) is used to estimate camera trajectory, then tracking speed is improved (500-1000 Hz), but the ability to attach virtual content to physical objects deteriorates because VIO does not relate to actual video stream content

Engineering Contradiction:

Improvetracking speedVSAvoidobject attachment reliability

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The patent combines VIO and OT into a fused tracker that processes video frames in both coordinate systems simultaneously. The VIO provides high-speed tracking while OT provides accurate object-model correspondence, and their results are merged through transform estimation to achieve both speed and reliable object attachment.

Inventive Principle:

Principle #5Merging (Combining)

2Reliability

If Object Tracking (OT) is used to track physical objects and attach virtual content, then object attachment reliability is improved, but tracking speed deteriorates significantly compared to VIO

Engineering Contradiction:

Improveobject attachment reliabilityVSAvoidtracking speed

Core Design Contradiction:

ReliabilityVSSpeed

Solution Approach 1:

The fused tracker merges OT's reliable object attachment capability with VIO's high-speed tracking. By processing frames in both coordinate systems and estimating the transform between them, the system achieves OT-level attachment reliability at VIO-level speeds.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The system performs preliminary VIO tracking at high speed for all frames, then uses OT results only when objects are detected to correct and refine the tracking. This allows the system to maintain high speed while periodically updating with reliable OT data.

Inventive Principle:

Principle #10Preliminary action

3Reliability

If Object Tracking (OT) is used to track objects in video streams, then virtual content can be anchored to physical objects, but the system must be restarted frequently when objects are not visible in the video stream

Engineering Contradiction:

Improveobject tracking stabilityVSAvoidtracking interruption time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

VIO acts as an intermediary that maintains continuous tracking even when OT cannot detect objects. The VIO trajectory serves as a bridge, allowing the system to maintain tracking state during object occlusions and only restart OT when needed, reducing interruptions.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system maintains preliminary VIO tracking continuously in the background, so when objects become visible again after occlusion, the tracking is already warmed up and ready to resume OT without full restart, reducing time loss.

Inventive Principle:

Principle #10Preliminary action

4Reliability

If a fused tracker combining VIO and OT is implemented, then tracking speed and robustness are improved, but system complexity increases due to processing in multiple coordinate systems

Engineering Contradiction:

Improvetracking robustnessVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system segments the tracking problem into two independent coordinate system processing streams (VIO and OT) that can be handled separately and then combined. This modular segmentation reduces overall complexity compared to a fully integrated approach.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The transform estimation between coordinate systems acts as an intermediary layer that bridges VIO and OT results. This intermediary approach simplifies the fusion process by providing a clear mathematical interface between the two coordinate systems without requiring complex direct integration.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS10832417B1Fusion of visual-inertial-odometry and object tracker for physically anchored augmented reality

Publication Date: 2020.11.10 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Embodiments may provide a fusion of Visual-Inertial-Odometry with Object Tracking for physically anchored Augmented Reality content presentation. For example, a method of generating augmented reality content may comprise receiving a video stream, comprising a plurality of frames, the video stream generated using a camera that is moving about a scene, determining a VIO camera pose for each frame in the video stream in a VIO coordinate system using visual inertial odometry, determining a OT camera pose for at least some of the frames of the video stream in a OT coordinate system using object tracking, determining a relation between the VIO coordinate system and the OT coordinate system and generating augmented reality content that is placed in the video stream of the scene at a location based on the determined relation between the VIO coordinate system and the OT coordinate system.