Metric Relative Pose Estimation from Image Keypoint Correspondences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining the relative pose between two images lack the ability to scale the pose to real-world metrics, necessitating additional depth measurements or contextual information, which can be unavailable or difficult to obtain, leading to inaccuracies in augmented reality applications.
Innovation Solution
A keypoint matching pipeline using a neural network estimates metric relative pose in 3D camera space without requiring depth measurements, training with supervised image pairs to determine scaled correspondences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth data is collected separately (e.g., from LiDAR) to obtain metric scale, then measurement precision is improved, but device complexity increases and depth data may be unavailable or difficult to obtain
Solution Approach 1:
The patent combines keypoint detection and depth estimation into a unified neural network model that processes image pairs directly to output metric relative poses. This integration eliminates the need for separate depth sensors like LiDAR, resolving the contradiction by achieving metric precision through computational methods rather than additional hardware
Solution Approach 2:
The patent replaces physical depth measurement mechanisms (LiDAR sensors) with a computational neural network model that infers metric depth relationships from image correspondences. This substitution achieves the same measurement function through software-based geometric reasoning rather than hardware-based optical measurement
2Adaptability or versatility
If depth estimation models are trained without contextual information, then adaptability is improved, but measurement precision deteriorates due to poor performance in areas with depth discontinuities
Solution Approach 1:
The patent employs local quality by detecting and utilizing depth discontinuities as explicit training signals. The model learns to associate visual features at discontinuity boundaries with depth changes, enabling it to handle complex scenes with varying depth characteristics while maintaining both adaptability and precision across diverse environments
3Adaptability or versatility
If keypoint detection is separated from depth estimation, then adaptability is improved, but measurement precision deteriorates due to inability to scale pose to real world
Solution Approach 1:
The patent merges keypoint detection and depth estimation into a single neural network that simultaneously outputs both detection results and metric depth information. This unified approach ensures that keypoints are detected with robustness while their depth relationships are estimated with metric accuracy, eliminating the scaling problem that arises when these functions are separated
Data Source
AI summary
A method for determining a metric relative pose between a target image and a reference image is disclosed. The method includes receiving the target image depicting a scene captured by a camera assembly of a client device. The method includes applying a machine-learning model to the target image to determine a metric relative pose between the target image and a reference image, wherein the metric relative pose represents a transformation from a pose of the reference image to a pose of the target image that is scaled to physical dimensions of the scene. The machine-learning model may include a keypoint network for determining a keypoint distribution including the spatial coordinates of keypoints extracted from each image. The machine-learning model may establish correspondences between the keypoint distribution of the target image and the keypoint distribution of the reference image. Based on the identified correspondences, the machine-learning model may regress the metric relative pose. With the metric relative pose, augmented reality content may be generated and displayed informed by the physical dimensions of the scene described by the metric relative pose.