Mixed Reality Video Rendering via Virtual Camera Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mixed reality systems that incorporate head-mounted displays (HMDs) require high computational resources to generate immersive experiences for both the user and spectators, making it challenging to render rich interactive experiences with reduced computational resources.
Innovation Solution
The implementation of a multi-pass processing technique that captures and adjusts the game play point of view (POV) to align with a camera POV, allowing for the creation of mixed reality videos by combining the user's game play performance with the in-game interaction view, using metadata to replay and composite the videos, and optionally using a 3D avatar for the player.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If real-time rendering of mixed reality video is performed by simultaneously generating multiple game views and processing player video, then immersive spectator experience is achieved, but computational resource consumption increases significantly
Solution Approach 1:
The system performs a calibration process before actual game playback to determine the spatial relationship between the camera and HMD. This pre-computed transformation matrix is stored and reused during spectator mode, avoiding the need for real-time coordinate transformations and view alignment calculations, thus reducing computational load during resource-constrained operation
Solution Approach 2:
Instead of rendering the game environment from multiple perspectives simultaneously, the system captures a single game view from the HMD and uses the pre-computed transformation to virtually reposition and reorient this single view to match the camera's perspective. This copying and transforming approach replaces the need for multiple independent rendering operations, significantly reducing GPU and CPU resource consumption
2Productivity
If multiple video processing operations are performed simultaneously including view generation, video capture, and compositing, then mixed reality video is produced, but processing time increases
Solution Approach 1:
The calibration data including camera position, orientation, and transformation matrices is computed and stored before spectator mode activation. During actual video generation, the system only needs to apply these pre-computed transformations to the captured HMD video stream, rather than performing complex geometric calculations in real-time, thus reducing processing time
Solution Approach 2:
The video processing pipeline is divided into distinct stages: capture phase (recording HMD and camera video), calibration phase (computing transformation relationships), and compositing phase (combining videos with background removal). By segmenting these operations and pre-computing transformation data, the system avoids performing all operations simultaneously, reducing peak processing time and enabling efficient real-time or near-real-time playback
3Ease of operation
If game view is rendered from player's POV without adjustment, then user immersion is maintained, but alignment with external camera view deteriorates
Solution Approach 1:
The system introduces a virtual camera as an intermediary element. This virtual camera is positioned and oriented according to the pre-computed transformation matrix that maps the relationship between the physical camera and the HMD's coordinate system. The game view is re-rendered from this virtual camera's perspective, automatically achieving alignment with the external camera while preserving the first-person gameplay experience for the spectator
Solution Approach 2:
The system dynamically adjusts the virtual camera parameters (position, orientation, field of view) based on the pre-computed transformation data. By changing these camera parameters rather than modifying the fundamental first-person perspective, the system achieves view alignment with the external camera while maintaining the immersive first-person gameplay experience, resolving the contradiction between immersion and alignment accuracy
Data Source
AI summary
A method is provided for rendering a mixed reality video. The method includes operations for capturing a head mounted display (HMD) game play by a user of a video game that is being executed on a computing system where the HMD game play is being captured from game play point of view (POV). The method further includes operations for identifying, by the computing system, a coordinate location of a camera that has a camera POV used to view the user during the HMD game play. In addition, the method further includes replaying the HMD game play to adjust the game play POV so that it substantially aligns with the camera POV. Moreover, the method includes rendering the mixed reality video by compositing video from the HMD game play after adjusting the game play POV and video from the camera POV. Rendering the mixed reality video also includes removing the background captured in the video from the camera POV so that the user appears partially within a scene of the video game when rendered in the mixed reality video.