AR Object Tracking via 3D Model Segmentation and Multi-View Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality (AR) object tracking methods face challenges in accurately recognizing and tracking 3D objects, especially those with complex shapes, due to high computational loads and inaccuracies in estimating position and gaze direction without a separate depth camera, leading to difficulties in implementing precise object tracking involving rotation and translation.
Innovation Solution
A method and system that simplify data processing for AR-based object tracking by obtaining a 3D definition model from images, learning from multiple viewpoints, updating the model, and extracting descriptors to minimize occlusion areas, using a combination of deep learning and 6 degrees of freedom (DoF) parameters to improve tracking efficiency and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If grid-based convolution process is performed for 3D object recognition, then local and global characteristics can be determined, but computational load increases substantially
Solution Approach 1:
The patent segments the 3D object recognition process into two distinct phases: (1) a coarse recognition phase using low-resolution depth maps to identify potential target regions, and (2) a fine recognition phase using high-resolution images only for the segmented regions of interest. This segmentation avoids applying computationally intensive grid-based convolution to the entire 3D space, thereby reducing overall computational load while maintaining recognition accuracy.
Solution Approach 2:
The patent introduces a depth dimension to create a 3D point cloud representation from 2D images and depth maps. By transforming the recognition problem into 3D space with explicit depth information, the system can perform region segmentation and feature extraction more efficiently, avoiding the need for exhaustive grid-based convolution across all spatial dimensions while preserving accurate spatial relationships.
2Reliability
If conventional SLAM technology is used for positioning, then virtual information can be placed in learned 3D space, but accuracy deteriorates in areas other than learned space
Solution Approach 1:
The patent performs preliminary action by pre-segmenting the 3D space into multiple regions of interest before object tracking begins. These pre-defined regions store spatial relationships and contextual information about different areas. When tracking occurs, the system can quickly reference these pre-prepared spatial models to accurately estimate object positions and orientations even in areas not previously learned through SLAM, thereby extending accurate positioning beyond the limitations of conventional SLAM.
Solution Approach 2:
The patent creates simplified copies or representations of 3D space regions (spatial models) that capture essential geometric and contextual features. These copied spatial representations are stored and can be rapidly matched against sensor data to estimate object positions. This copying approach allows the system to generalize position estimation to new areas without requiring complete prior learning of those spaces through traditional SLAM methods.
3Device complexity
If single general-purpose camera is used for capturing images, then device complexity is reduced, but accuracy of estimating position and gaze direction deteriorates
Solution Approach 1:
The patent changes the parameters being measured by the single camera system. Instead of relying on the camera to directly provide accurate depth and position data, the system processes the camera images through multiple transformations: converting to grayscale, applying adaptive thresholding, performing morphological operations, and extracting features at multiple scales. These parameter transformations enable the single camera to effectively estimate position and gaze direction with accuracy comparable to multi-camera systems.
Solution Approach 2:
The patent introduces an intermediary processing pipeline between the single camera and the object tracking function. This intermediary includes depth map generation, region segmentation, and feature extraction stages that mediate the information from the single camera. These intermediary processing steps extract and enhance positional and orientational information that would otherwise be unavailable from a single general-purpose camera, enabling accurate tracking without additional hardware.
4Productivity
If deep learning methods are used for 3D object recognition, then object tracking can be performed, but computational load increases for complex shapes
Solution Approach 1:
The patent segments both the 3D space and the object processing into coarse and fine stages. In the coarse stage, low-resolution depth maps are processed to identify candidate regions and basic object locations. In the fine stage, high-resolution images are processed using deep learning methods only for the segmented regions of interest. This spatial and resolution-based segmentation reduces the amount of data that requires computationally intensive deep learning processing, thereby reducing overall computational load while maintaining tracking capability.
Solution Approach 2:
The patent applies deep learning methods partially rather than exhaustively. Instead of processing all image data at full resolution through deep learning networks, the system applies these computationally intensive methods only to selected regions identified in a preliminary coarse processing stage. This partial application of deep learning provides sufficient tracking performance for complex shapes while significantly reducing the computational burden compared to full-resolution processing.
Data Source
AI summary
An object tracking method for augmented reality according to an embodiment of the present disclosure, by which a tracking application executed by at least one processor of a terminal performs object tracking for augmented reality, comprises obtaining a 3D definition model trained based on images capturing a target object from a first viewpoint; performing object tracking of the target object based on the obtained 3D definition model; obtaining a plurality of frame images from a plurality of viewpoints for the target object based on the object tracking; learning the target object from the plurality of viewpoints based on the plurality of frame images obtained; updating the 3D definition model based on the learning; and performing AR object tracking for the target object based on the updated 3D definition model.