HMD Camera View Alignment Using Assembled Frames for Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current telepresence systems using head-mounted displays (HMDs) for remote user control of movable cameras face challenges in matching the orientation of the camera view with the user's head motion, leading to motion sickness due to latency issues and mismatch between the user's inner ear sense of orientation and the changing camera view.
Innovation Solution
A system that includes a head-mounted display with a head position tracker and a camera position controller, which generates a head position signal to align the camera's position with the user's head movement, using a surface map of past views to assemble and render a 3D image that appears to match the user's head motion in real-time, eliminating delays and ensuring seamless alignment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the HMD displays the camera view while the user controls the drone with a joystick, then the user can operate the drone remotely, but the user experiences motion sickness due to mismatch between head orientation and displayed view orientation
Solution Approach 1:
The system uses a head position tracker to continuously monitor the user's head orientation and feeds this information back to the camera position controller, which automatically adjusts the camera view orientation to match the user's head movement, creating a closed-loop feedback system that eliminates the orientation mismatch causing motion sickness
Solution Approach 2:
The system creates a virtual copy of the user's head position and orientation and applies it to the camera view display on the HMD, so the displayed view orientation perfectly replicates what the user would see if they were physically present at the camera location, eliminating the sensory conflict that causes motion sickness
2Measurement precision
If the HMD tracks and displays the camera view aligned with user head position, then the view orientation matches head motion, but latency causes mismatch between inner ear sense and displayed view
Solution Approach 1:
The system performs preliminary actions by predicting the user's future head position based on current head movement velocity and acceleration data from the head position tracker, and pre-adjusts the camera view orientation to match the predicted position, compensating for processing and display latency before the mismatch occurs
Solution Approach 2:
The system dynamically adjusts the camera view orientation in real-time by continuously updating the head position tracking data and recalculating the required camera rotation angles, allowing the display to adapt to changing head movements and maintain synchronization despite varying speeds and directions of motion
3Adaptability or versatility
If the camera view is continuously adjusted to match head movement, then the telepresence experience is immersive, but the system complexity increases due to additional sensors and processing
Solution Approach 1:
The HMD device performs multiple functions by integrating both the camera view display and the head position tracking capabilities into a single device, allowing it to both capture the user's head orientation and display the corresponding camera view, reducing the need for separate external tracking devices and simplifying the overall system architecture
Solution Approach 2:
The system merges the head position tracking function with the camera view rendering function by directly coupling the tracker output to the camera position controller, eliminating intermediate processing stages and reducing system complexity while maintaining the immersive telepresence experience
Data Source
Figure 1
Figure 2~3
Figure 4A~4C
AI summary
Tracking a user head position detects a change to a new head position and, in response, a remote camera is instructed to move to a next camera position. A camera image frame, having an indication of camera position, is received from the camera. Upon the camera position not aligning with the next camera position, an assembled image frame is formed, using image data from past views, and rendered to appear to the user as if the camera moved in 1:1 alignment with the user's head to the next camera position.