AR Pose Prediction via IMU and Camera Fusion

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing augmented reality (AR) devices face challenges in providing real-time output images that accurately reflect the pose of the wearable device, leading to user discomfort due to discrepancies between detected poses and output images.

Innovation Solution

The method involves using an inertial measurement unit (IMU) sensor and a camera in the AR device to obtain IMU values and images at different frequencies. These values are then used to estimate intermediate 6-degrees of freedom (6D) poses, which are fed into a deep neural network to generate a pose prediction model for predicting relative 6D poses.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If IMU sensor and camera are used to obtain data at different frequencies, then measurement precision of pose is improved, but device complexity increases

Engineering Contradiction:

Improvepose detection accuracyVSAvoidsensor integration complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The system segments the pose detection task into two independent data streams: IMU sensor data for high-frequency motion capture and camera data for visual pose estimation. Each sensor type processes its own data independently before fusion, reducing integration complexity while maintaining high measurement precision through multi-source data combination.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A deep neural network acts as an intermediary that fuses IMU values and camera images to generate accurate pose predictions. The DNN model integrates data from different frequency sources and reconciles their differences, achieving high pose detection accuracy without requiring direct complex hardware integration between sensors.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If deep neural network is used for pose prediction, then productivity of real-time processing is improved, but use of energy increases

Engineering Contradiction:

Improvereal-time processing speedVSAvoidprocessor energy consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary processing by extracting key features from IMU values and camera images before feeding them to the deep neural network. Intermediate 6D poses are estimated in advance using visual-inertial SLAM, reducing the computational burden on the DNN during real-time operation and lowering energy consumption while maintaining high processing speed.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The deep neural network processes only the most critical features and intermediate pose estimates rather than all raw sensor data. By selectively processing partial data that contains the most relevant information for pose prediction, the system achieves real-time processing performance with reduced energy consumption compared to processing complete data streams.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS12288419B2Augmented reality (AR) device and method of predicting pose therein

Publication Date: 2025.04.29 SAMSUNG ELECTRONICS CO LTD

AI summary

An augmented reality (AR) device and a method of predicting a pose in the AR device is provided. In the augmented reality (AR) device inertial measurement unit (IMU) values corresponding to the movement of the AR device are obtained at an IMU rate, intermediate 6-degrees of freedom (6D) poses of the AR device are estimated based on the IMU values and images around the AR device via a visual-inertial simultaneous localization and mapping (VI-SLAM) module, and a pose prediction model for predicting relative 6D poses of the AR device is generated by performing learning by using a deep neural network.