Video Conference Gaze Retargeting for Accurate Attention Cues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In video conferences, participants lose the visual feedback of who others are paying attention to, as gaze direction is not accurately represented across different displays, leading to a lack of social connection and clarity in virtual environments.
Innovation Solution
The system detects and manipulates gaze direction by generating metadata to retarget a participant's gaze towards the display area of the target participant, using modules for gaze understanding, metadata generation, and gaze retargeting to ensure accurate gaze representation across multiple displays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If gaze direction is captured in video conference, then visual feedback of attention is improved, but gaze direction accuracy across different displays deteriorates
Solution Approach 1:
The system transforms gaze direction parameters from the original camera coordinate system to a display-specific coordinate system. By detecting the target participant's identity and calculating the corresponding display area position, the gaze direction parameters are dynamically adjusted to match each participant's unique display layout, thereby maintaining gaze accuracy across different displays.
Solution Approach 2:
The system introduces an intermediary processing layer that includes a gaze understanding module, metadata generation module, and gaze retargeting module. This intermediary layer translates raw gaze data into retargeted gaze directions that are adapted to each participant's display configuration, mediating between the original gaze capture and the final display presentation.
2Loss of information
If video signal is transmitted to multiple participants, then visual feedback is provided to all, but gaze direction becomes misaligned with target participant display area
Solution Approach 1:
The system dynamically adjusts gaze direction for each participant based on their specific display layout and the target participant's identity. Instead of using a static gaze direction, the system continuously retargets the gaze to match the dynamic display configuration of each participant, ensuring that the gaze always points to the correct display area regardless of how the video layout changes.
Solution Approach 2:
The system applies different gaze retargeting parameters to different participants based on their local display characteristics. Each participant receives a customized gaze direction that is optimized for their specific display area where the target participant's video is shown, rather than using a universal gaze direction for all participants.
Data Source
AI summary
A method at a first participant's client conferencing system in a videoconference comprises receiving, from a second client conferencing system, at least one first video frame of a first video signal including an image of the second participant looking at a third participant, and first metadata associated with the first video frame and including an identity of the third participant. The image of the second participant is modified in the first video frame so that the first video frame is displayed on a first area of the client conferencing system with the second participant looking at a second area of the first display configured for displaying a second video signal of the third participant identified by the first metadata.