Real-Time AR Video Mixing Using Sensor-Based Camera Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality systems for TV broadcast, cinema, and video games rely heavily on optical flow analysis or marker-based approaches, which are often unreliable and inefficient, particularly in dynamic environments like movie production.
Innovation Solution
A system that uses sensors attached to a film camera to provide real-time 3D positioning data, allowing for the automatic creation, rendering, or modification of computer-generated 3D objects and their real-time mixing with a video feed from the camera.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If optical flow analysis is used to track camera position and orientation, then the system can operate without physical markers, but the tracking reliability deteriorates in dynamic environments
Solution Approach 1:
The patent combines multiple tracking methods (optical flow analysis and marker-based tracking) into a unified system that automatically selects or switches between them based on environmental conditions, thereby maintaining both the ease of markerless operation and the reliability of marker-based tracking when needed
Solution Approach 2:
The patent introduces virtual markers or reference objects that are digitally overlaid onto the scene, serving as intermediaries that provide reliable tracking points without requiring physical markers to be placed in the environment, thus bridging the gap between markerless operation and reliable tracking
2Reliability
If marker-based approaches are used for tracking, then tracking reliability improves, but setup time and complexity increase significantly
Solution Approach 1:
The patent uses virtual copies or digital representations of markers that can be programmatically placed in the scene without physical setup, providing the tracking reliability of marker-based systems while eliminating the time-consuming physical marker placement process
Solution Approach 2:
The patent replaces the mechanical process of physically placing and positioning markers with a digital/computational system that automatically generates and positions virtual markers through software, thereby maintaining tracking reliability while eliminating setup time
3Device complexity
If pure optical flow analysis is used, then setup complexity is reduced, but tracking accuracy deteriorates
Solution Approach 1:
The patent applies different tracking strategies to different regions or conditions of the scene, using optical flow analysis in simple, stable regions and switching to marker-based or enhanced tracking methods in complex or dynamic regions, thereby maintaining low overall complexity while improving accuracy where needed
Data Source
AI summary
The system generates real-time augmented reality video for TV broadcast, cinema or video games. The system includes a monoscopic video camera including a body, a stereoscopic video camera, and a processor. The system includes sensors, including multiple non-optical sensors, which provide real-time positioning data defining the 3D position and 3D orientation of the monoscopic video camera, or enable the 3D position and 3D orientation of the monoscopic video camera to be calculated. The processor is configured to use the real-time positioning data automatically to create, recall, render or modify computer generated 3D objects. The processor is configured to determine the 3D position and orientation of the monoscopic video camera with reference to a 3D map of the real-world generated whilst the camera is being used to capture video. The processor is configured to track the scene without a requirement for an initial or prior survey of the scene.