AR Video Generation With Motion-Tracked Virtual Camera Backgrounds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video creation tools fail to account for camera motion, limiting the quality and realism of background visual effects and integration of virtual foreground objects, as they do not leverage the sophisticated motion tracking capabilities of modern mobile devices.
Innovation Solution
Systems and methods that integrate complex computer-generated backgrounds and foregrounds, adjusting the rendered scene based on the position and orientation of a physical camera, using AR applications to track and update virtual camera positions and orientations in real-time, leveraging inertial measurement and chroma key technology.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If simple one-to-one pixel replacement is used for chroma key background replacement, then the implementation is simple and fast, but the visual effect quality is poor and does not account for camera motion
Solution Approach 1:
The patent applies dynamics by transforming the static background replacement process into a dynamic one that responds to camera motion. The system continuously tracks camera position and orientation using inertial sensors and updates the rendered background in real-time to match the camera's movement, creating a dynamic visual effect that maintains realism throughout the video recording.
Solution Approach 2:
The system implements feedback by using inertial measurement unit (IMU) data to continuously monitor camera motion and feed this information back to the background rendering process. The rendered background is adjusted based on the detected camera position and orientation changes, creating a closed-loop system that maintains visual consistency between the captured foreground and generated background.
2Productivity
If the computer-generated background stays fixed within the camera's frame of reference, then the processing is simple, but the realism is poor as it does not update with camera position changes
Solution Approach 1:
The patent makes the background rendering dynamic by continuously updating it based on camera motion. Instead of a fixed background, the system renders new background frames that correspond to the camera's current position and orientation, creating a realistic effect where the background appears to move naturally with the camera throughout the video recording.
3Reliability
If AR effects are used to create virtual objects responsive to camera movement, then the realism is improved, but the complexity of background detection and replacement increases significantly
Solution Approach 1:
The patent extracts the background detection and replacement complexity from the AR processing pipeline. By using chroma key technology to separately identify and remove the background, the system simplifies the overall processing while still achieving realistic virtual object integration. This extraction allows virtual objects to be composited independently without requiring complex scene understanding.
Solution Approach 2:
The system uses an intermediary approach by introducing a virtual camera that mirrors the physical camera's motion. This virtual camera serves as a mediator between the physical camera's movement and the rendered background, translating real-world motion into corresponding virtual scene changes without requiring direct complex analysis of the captured footage.
Data Source
AI summary
Systems and methods are described for generating an AR image are described herein. A physical camera is used to capture a video of a physical object in front of a physical background. The system then accesses data defining a virtual environment and selects a first position of a virtual camera in the virtual environment. While capturing the video, the system displays captured video of the physical object, such that the physical background is replaced with a view of the virtual environment from the first position of the virtual camera. In response to detecting a movement of the physical camera, the system selects a second position of the virtual camera in the virtual environment based on the detected movement. The system then displays the captured video of the physical object, wherein the view of the physical background is replaced with a view of the virtual environment from the second position of the virtual camera.