Foreground Object Detection in Augmented Reality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality (AR) interaction methods are limited and non-intuitive, as users rely on physical input elements or virtual buttons, which do not allow natural interaction with virtual objects, especially under partial occlusions.
Innovation Solution
A method and apparatus that detect and track a movable foreground object, such as a user's fingers, in captured images without depth information, by warping and comparing images to segment and render augmentations, enabling natural interaction with AR objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If virtual buttons are used for interaction, then user interaction with AR objects is enabled, but the interaction feels non-intuitive and unnatural
Solution Approach 1:
The patent replaces physical buttons and traditional virtual button interfaces with direct hand gesture recognition. The system uses computer vision to detect and track hand movements, allowing users to interact with AR objects through natural hand motions rather than mechanical or virtual button interfaces. This substitution of mechanical interaction systems with vision-based gesture recognition enables intuitive and natural user interaction.
2Measurement precision
If pattern-based AR systems are used, then accurate registration of virtual objects is achieved, but robustness to partial occlusions is insufficient
Solution Approach 1:
The patent segments the scene into foreground (hand) and background (AR pattern) components using image processing techniques. By separating the hand detection from the pattern recognition processes, the system can accurately track hand gestures even when the hand partially occludes the AR pattern. This segmentation allows independent optimization of both pattern registration accuracy and hand tracking robustness.
Solution Approach 2:
The patent introduces an intermediary background subtraction process that compares current frames with reference frames to identify foreground hand movements. This intermediary step acts as a mediator between the raw image data and the final gesture recognition, enabling the system to distinguish hand occlusions from pattern features and maintain reliable tracking under partial occlusion conditions.
3Ease of operation
If hand tracking is implemented, then natural interaction with virtual objects is achieved, but detection accuracy under occlusion is reduced
Solution Approach 1:
The patent performs preliminary background modeling by capturing reference frames before hand interaction begins. This preliminary action creates a baseline representation of the scene without hands, which is then used to subtract background elements from subsequent frames. By preparing this reference background in advance, the system maintains high hand detection accuracy even when hands occlude significant portions of the scene, as the background subtraction isolates hand features from occluded pattern areas.
Data Source
Figure 1
Figure 2~3A
Figure 3B~4
AI summary
A mobile device detects a moveable foreground object in captured images, e.g., a series of video frames without depth information. The object may be one or more of the user's fingers. The object may be detected by warping one of a captured image of a scene that includes the object and a reference image of the scene without the object so they have the same view and comparing the captured image and the reference image after warping. A mask may be used to segment the object from the captured image. Pixels are detected in the extracted image of the object and the pixels are used to detect the point of interest on the foreground object. The object may then be tracked in subsequent images. Augmentations may be rendered and interacted with or temporal gestures may be detected and desired actions performed accordingly.