Photorealistic Avatar Animation for Gaze-Aware Virtual Conferencing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In virtual conferences, participants' gaze directions and speech are not accurately conveyed, leading to difficulties in determining who is looking at whom and issues with user experience when cameras are turned off due to bandwidth or privacy concerns.
Innovation Solution
The use of machine learning models to generate synthesized images based on participant photos, combined with gaze and speech data, to animate photorealistic avatars in virtual conferencing systems, reducing the need for live video streaming and enhancing user experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If live video streaming is used to convey participant appearance and gaze, then accuracy of gaze and speech representation is improved, but bandwidth consumption increases and privacy concerns arise
Solution Approach 1:
The patent creates photorealistic avatar copies of participants that replicate their appearance, gaze directions, and speech characteristics. These avatars are synthesized from static photos and audio recordings, allowing the system to convey participant presence without streaming live video, thus reducing bandwidth consumption while maintaining representation accuracy
Solution Approach 2:
The system performs preliminary actions by capturing static photos and audio recordings of participants before the conference, then uses these pre-captured materials to synthesize avatars. This preliminary capture eliminates the need for continuous live video streaming during the conference, reducing bandwidth consumption while preserving the ability to accurately represent participant gaze and speech
2Reliability
If cameras are turned off to protect privacy and reduce bandwidth usage, then privacy is protected and bandwidth is reduced, but user experience deteriorates due to loss of visual presence
Solution Approach 1:
The patent creates photorealistic avatar copies of participants that replicate their appearance, gaze directions, and speech characteristics. These avatars are synthesized from static photos and audio recordings, allowing the system to convey participant presence without streaming live video, thus reducing bandwidth consumption while maintaining representation accuracy
Solution Approach 2:
The photorealistic avatars serve as intermediaries between the actual participants and other conference attendees. The avatars convey participant presence, gaze, and speech characteristics without requiring participants to activate cameras, thus mediating the interaction while protecting privacy and reducing bandwidth usage
3Measurement precision
If multiple synthesized images are generated for different gaze directions, then accuracy of gaze representation is improved, but computational complexity increases
Solution Approach 1:
The system performs preliminary actions by generating multiple synthesized images with different gaze directions in advance, before the conference begins. These pre-synthesized images are stored and then selected based on the participant's actual gaze direction during the conference, reducing real-time computational complexity while maintaining gaze representation accuracy
Solution Approach 2:
The system dynamically selects and blends appropriate synthesized images based on the participant's current gaze direction. By using interpolation and blending techniques, the system can generate intermediate gaze directions from discrete synthesized images, achieving continuous gaze representation without synthesizing images for every possible angle, thus balancing accuracy with computational efficiency
Data Source
Figure 1A
Figure 1B
Figure 1C~1D
AI summary
According to a general aspect, a method can include receiving a photo of a virtual conference participant, and a depth map based on the photo, and generating a plurality of synthesized images based on the photo. The plurality of synthesized images can have respective simulated gaze directions of the virtual conference participant. The method can also include receiving, during a virtual conference, an indication of a current gaze direction of the virtual conference participant. The method can further include animating, in a display of the virtual conference, an avatar corresponding with the virtual conference participant. The avatar can be based on the photo. Animating the avatar can be based on the photo, the depth map and at least one synthesized image of the plurality of synthesized images, the at least one synthesized image corresponding with the current gaze direction.