AR Video Camera Pose Estimation Using Filtered Image Matches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality (AR) technologies face challenges in achieving smooth and realistic camera tracking without the use of markers, as marker-based methods require setup time and markerless methods are prone to errors and jerky movements.
Innovation Solution
A hybrid approach using a deep learning-based image matcher and a match refinement procedure, combined with a perspective-n-point (PnP) pose computation, to estimate camera pose based on high-stability point-to-point matches, eliminating the need for markers and improving tracking stability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If marker-based methods are used for camera tracking, then tracking stability is improved, but setup time and device complexity increase
Solution Approach 1:
The patent extracts and removes the marker components from the tracking system, transitioning from marker-based to markerless AR. This eliminates the need for physical markers while maintaining tracking functionality through direct feature detection and matching between template and content images, thereby reducing setup time while preserving tracking stability through algorithmic improvements
2Loss of time
If markerless methods are used for camera tracking, then setup time is reduced, but tracking accuracy and smoothness deteriorate due to errors and jerky movements
Solution Approach 1:
The patent replaces traditional marker-based mechanical tracking systems with a deep learning-based image matching system. A trained convolutional graph neural network performs feature detection and matching, substituting physical marker recognition with automated neural network-based feature correspondence, thereby achieving both markerless operation and high tracking precision
Solution Approach 2:
The patent changes the parameters of feature matching by using a trained neural network to evaluate match quality and apply stability thresholds. This transforms the feature matching process from simple geometric comparison to a learned parameter-based evaluation, improving accuracy by filtering out unreliable matches and retaining only high-stability correspondences for pose computation
3Productivity
If traditional feature matching is used, then computational speed is improved, but matching accuracy and reliability worsen due to false matches
Solution Approach 1:
The patent introduces a trained convolutional graph neural network as an intermediary between feature detection and pose computation. This neural network acts as a mediator that evaluates feature match quality, applies stability thresholds, and filters matches before they are used for camera pose estimation, thereby improving reliability without significantly compromising processing speed
Data Source
AI summary
Certain aspects and features of this disclosure relate to providing a hybrid approach for camera pose estimation using a deep learning-based image matcher and a match refinement procedure. The image matcher takes an image pair as an input and estimates coarse point-to-point feature matches between the two images. The coarse point-to-point feature matches can be filtered based on a stability threshold to produce high-stability point-to-point matches. A perspective-n-point (PnP) camera pose for each frame of video, including one or more added digital visual elements can be computed using the high-stability matches and video frames can be rendered, each using its computed camera pose.