AR Object Placement via Image Segmentation and Pose Interpolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Augmented reality applications face challenges in seamlessly merging virtual objects with real environments to provide a satisfactory user experience, particularly in maintaining the relative position and orientation of augmented reality objects as the device moves, and in preventing objects from 'jumping' when pose tracking is lost.
Innovation Solution
An electronic device with a camera and pose sensors captures images of the real environment, segments them into foreground and background pixels, and composites augmented reality objects between these layers, adjusting their pose based on device movement and interpolating poses when tracking is regained to maintain a stable appearance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If augmented reality objects are placed in images based on device pose tracking, then the immersive integration and user experience are improved, but the objects may jump or become unstable when pose tracking is lost
Solution Approach 1:
The system performs preliminary pose interpolation when pose tracking is lost, using the last known valid pose and current device position to calculate intermediate poses. This preliminary action prevents the abrupt jumping of augmented reality objects by smoothly transitioning between known poses, thereby maintaining stability without requiring complex real-time tracking during the interpolation period.
Solution Approach 2:
The patent introduces pose interpolation as an intermediary mechanism between direct pose tracking and augmented reality object placement. When tracking is lost, this intermediary process calculates virtual poses based on device motion data, serving as a bridge that maintains object stability without directly relying on the failed tracking system, thus resolving the contradiction between reliability and tracking complexity.
2Reliability
If the system continuously tracks and adjusts pose to maintain object position, then the immersive experience is improved, but the processing requirements and computational load increase
Solution Approach 1:
The system implements periodic pose updates rather than continuous real-time adjustments. Pose tracking and adjustment occur at discrete intervals based on available device data, reducing computational load and energy consumption while maintaining sufficient object position consistency. This periodic approach balances reliability requirements with energy efficiency by updating only when necessary.
Solution Approach 2:
The pose interpolation system uses readily available device motion data and previously stored pose information to self-generate corrective poses without requiring additional complex sensing or processing. This self-service mechanism maintains object position consistency using existing system resources, minimizing additional energy consumption while achieving the desired reliability.
3Ease of operation
If the augmented reality object maintains fixed relative position to the device, then the user experience is improved, but the object may appear to jump when the device moves rapidly or tracking is lost
Solution Approach 1:
The system dynamically adjusts the relationship between device pose and augmented reality object placement by switching between direct tracking mode and interpolation mode. When tracking is reliable, objects maintain fixed relative positions for good user experience. When tracking degrades or device moves rapidly, the system transitions to dynamic interpolation that adapts to the new conditions, preventing visual jumps while maintaining operational ease.
Solution Approach 2:
The patent combines multiple pose estimation approaches (direct tracking, interpolation, and device motion data) into a composite pose system. This composite approach integrates the strengths of each method: direct tracking provides accuracy when available, while interpolation and motion data provide stability during rapid movement or tracking loss, thereby maintaining both user experience quality and visual stability without contradiction.
Data Source
AI summary
An electronic device places an augmented reality object in an image of a real environment based on a pose of the electronic device and based on image segmentation. The electronic device includes a camera that captures images of the real environment and sensors, such as an inertial measurement unit (IMU), that capture a pose of the electronic device. The electronic device selects an augmented reality (AR) object from a memory, segments a captured image of the real environment into foreground pixels and background pixels, and composites an image for display wherein the AR object is placed between the foreground pixels and the background pixels. As the pose of the electronic device changes, the electronic device maintains the relative position of the AR object with respect to the real environment in images for display.