Separated-Camera Optical Alignment Using Pose-Based Late Reprojection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional mixed-reality systems face challenges in aligning image content from multiple cameras due to the lack of timestamp or pose data, especially when cameras operate in different time domains or are remotely located, which affects hologram placement and generation.
Innovation Solution
The system aligns images generated by an integrated camera mounted to a computer system with those from a detached camera using late stage reprojection (LSR) and inertial measurement units (IMUs) to account for pose differences, enabling image alignment without relying on timestamp data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If timestamp data is required for image alignment, then alignment precision is improved, but system complexity increases and reliability decreases when cameras operate remotely or in different time domains
Solution Approach 1:
The patent introduces pose information as an intermediary element to bridge the gap between images captured at different times. Instead of directly relying on timestamp data for alignment, the system uses pose information (representing camera position and orientation) as a mediator to establish the spatial relationship between images from different time domains, thereby achieving reliable alignment without timestamp dependency
Solution Approach 2:
The patent changes the fundamental parameter used for alignment from timestamp-based temporal synchronization to pose-based spatial transformation. By transforming the alignment problem from the temporal domain (timestamps) to the spatial domain (pose matrices), the system can align images captured at different times reliably, resolving the contradiction between precision and reliability
2Adaptability or versatility
If pose information is used for alignment, then alignment capability without timestamps is improved, but device complexity increases
Solution Approach 1:
The patent makes the pose estimation module multi-functional by enabling it to serve both as a tracking mechanism and as an alignment tool. The same pose information derived from tracking camera position can be universally applied to align images from multiple cameras without requiring separate alignment hardware or complex additional systems, thereby improving adaptability without proportionally increasing complexity
3Adaptability or versatility
If multiple cameras operate independently, then system flexibility is improved, but image alignment difficulty increases
Solution Approach 1:
The patent segments the alignment process into independent steps: first estimating pose for each camera independently, then using these individual pose estimates to align images. This segmentation allows each camera to operate independently with its own pose estimation, while the overall alignment is achieved through a systematic combination of these independent results, reducing the difficulty of alignment
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Improved techniques for generating images are disclosed herein. A first image is generated by an integrated camera. The pose of the computer system is determined based on the image, and a timestamp is determined. A detached camera generates a second image. The second image is aligned with the first image. An overlaid image is generated by overlaying the second image onto the first image based on the alignment. A pose difference is then identified between a current pose of the computer and the initial pose. Consequently, late stage reprojection (LSR) is performed on the overlaid image to account for the pose difference. The LSR-corrected overlaid image is then displayed.