Gaze-Driven Virtual Camera Viewpoint Projection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for capturing and displaying video streams do not effectively replicate the view of a person within a scene, particularly in dynamic environments where the viewer's perspective changes.
Innovation Solution
A method that captures multiple video streams of a scene using one or more cameras, reconstructs a virtual environment, determines the gaze direction of a person within the scene, and projects the virtual environment onto a plane normal to the gaze direction for display from the person's viewpoint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple video streams are captured and a virtual environment is reconstructed to represent the scene, then the accuracy of representing the person's viewpoint is improved, but the device complexity and computational resources required increase
Solution Approach 1:
The system segments the complex task of viewpoint reconstruction by dividing it into distinct modules: multiple cameras capture different video streams, gaze detection separately identifies the person's line of sight, and a virtual environment is reconstructed by combining these elements. This segmentation allows each component to be optimized independently while achieving high overall accuracy.
Solution Approach 2:
A virtual camera is introduced as an intermediary element that mediates between the multiple physical cameras and the final displayed image. The virtual camera's position and orientation are dynamically adjusted based on detected gaze information, allowing the system to represent the person's viewpoint without requiring direct complex processing of all physical camera feeds at once.
2Adaptability or versatility
If the video stream is dynamically adjusted based on gaze direction, then the immersiveness of the visual experience is improved, but the processing speed and real-time performance may deteriorate
Solution Approach 1:
The system performs preliminary actions by pre-processing video streams from multiple cameras and maintaining ready-to-use virtual environment representations. When gaze direction is detected, the system can quickly adjust the virtual camera parameters and render the appropriate viewpoint without requiring extensive real-time computation, thus maintaining high processing speed while achieving adaptive viewpoint representation.
Solution Approach 2:
The system implements dynamic adjustment of the virtual camera's position and orientation based on real-time gaze detection. The virtual environment is rendered dynamically to match the person's changing viewpoint, creating an immersive experience. The system balances this dynamic adaptation with optimized rendering techniques to maintain real-time performance.
3Measurement precision
If gaze detection is used to determine the person's line of sight, then the accuracy of viewpoint determination is improved, but the difficulty of detecting and measuring gaze direction increases
Solution Approach 1:
The system uses an intermediary virtual camera as a reference frame to simplify gaze detection. Instead of directly analyzing complex eye movement patterns in the original video streams, the system detects gaze relative to the virtual camera's coordinate system, which is already aligned with the reconstructed virtual environment. This intermediary reference system reduces the complexity of gaze measurement while maintaining high accuracy.
4Measurement precision
If the virtual environment is projected onto a plane normal to gaze direction, then the accuracy of simulating the person's view is improved, but the computational complexity of rendering increases
Solution Approach 1:
The rendering system dynamically adjusts the projection plane orientation to be normal to the detected gaze direction. Instead of using a fixed projection plane, the system rotates and repositions the virtual camera's projection plane in real-time based on gaze detection results. This dynamic approach accurately simulates the person's viewpoint while using efficient rendering techniques to manage computational complexity.
Data Source
AI summary
The present invention relates to a method for capturing and displaying a video stream, comprising: capturing with one or a plurality of cameras a plurality of video streams of a scene, said scene comprising at least one person; reconstructing from said plurality of video streams a virtual environment representing the scene, determining the gaze direction of said person using at least one of said plurality of video streams; projecting said virtual environment onto a plane normal to said gaze direction for generating a virtual representation corresponding to what that person is looking at and from the point of view of that person; displaying said virtual representation on a display.