Video Clip Object Tracking with Sensor-Based AR Playback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Virtual objects in augmented reality experiences often disappear or behave erratically due to environmental conditions and unanticipated visual interruptions, breaking the illusion of their presence in real-world environments.
Innovation Solution
A system that captures sensor information during video recording and stores tracking indicia, allowing for dynamic adjustment of virtual object placement relative to real-world objects, using multiple redundant tracking systems to seamlessly transition between 6DoF and 3DoF, ensuring smooth rendering of virtual objects in previously captured videos.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional augmented reality systems are used to render virtual objects in real-world environments, then the illusion of virtual objects being present can be created, but environmental conditions and visual interruptions cause the virtual objects to disappear or behave erratically
Solution Approach 1:
The tracking system is divided into multiple independent tracking subsystems, each capable of tracking virtual objects using different methods. When one subsystem fails due to environmental conditions or visual interruptions, other subsystems can take over, ensuring continuous and stable rendering of virtual objects throughout the video clip.
Solution Approach 2:
The system pre-processes video clips to identify and mark keyframes where virtual objects are clearly visible and trackable. These pre-identified keyframes serve as anchor points that can be used to reconstruct object positions even when visual interruptions occur between frames, maintaining rendering stability.
2Adaptability or versatility
If virtual objects are added to video clips after capture, then user interaction and post-processing flexibility are improved, but tracking accuracy and object positioning may deteriorate due to lack of real-time feedback
Solution Approach 1:
The system uses sensor information captured during video recording (such as IMU data, GPS coordinates, and camera metadata) as feedback to continuously refine and adjust the positions and orientations of virtual objects added post-capture. This feedback mechanism compensates for the lack of real-time interaction, maintaining high tracking accuracy even when objects are added after video capture.
3Reliability
If sensor information is captured and stored with video clips, then post-capture rendering and tracking are enhanced, but data storage requirements and processing complexity increase
Solution Approach 1:
The system extracts and stores only the essential sensor information needed for tracking (such as device orientation, position, and motion data) separately from the video clip itself. This extracted tracking data is stored in a compact format that can be efficiently processed and applied during post-capture rendering, reducing overall data storage requirements while maintaining tracking consistency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for rendering a three-dimensional virtual object in a video clip. The method and system include capturing, using a camera-enabled device, video content of a real-world scene and movement information collected by the camera-enabled device during capture of the video content. The captured video and movement information are stored. The stored captured video content is processed to identify a real-world object in the scene. An interactive augmented reality display is generated that: adds a virtual object to the stored video content to create augmented video content comprising the real-world scene and the virtual object; and adjusts, during playback of the augmented video content, an on-screen position of the virtual object within the augmented video content based at least in part on the stored movement information.