Immersive Teleconferencing with Synthesized Orientation-Aware Avatars
Solution Overview
Problem
Current teleconferencing technologies primarily focus on symmetrical bidirectional videoconferencing or one-to-many broadcast, lacking immersive options that allow remote participants to view a real-time, immersive feed of a real-world scene while communicating with both people in the scene and other remote participants.
Innovation Solution
The system enables near real-time communication among real and virtual participants by using a host device to capture and transmit a wide-field-of-view audio-visual feed, allowing client devices to display real-world video or avatars of participants, and adjusting the visual representation based on orientation information to create an immersive experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If symmetrical bidirectional videoconferencing or one-to-many broadcast is used, then communication between participants is enabled, but immersive presence and real-time interaction in virtual environments are lacking
Solution Approach 1:
The system segments teleconferencing into distinct modes: symmetrical bidirectional videoconferencing for direct participant-to-participant communication, one-to-many broadcast for host-to-remote communication, and immersive teleconferencing for virtual environment experiences. Each mode uses appropriate technical configurations, allowing the system to adapt to different communication needs without requiring all features to be simultaneously complex.
Solution Approach 2:
The teleconferencing system is designed to perform multiple functions across different operational modes. The same basic infrastructure supports symmetrical videoconferencing, broadcast, and immersive teleconferencing with virtual reality headsets, making the system versatile while managing complexity through shared components and protocols.
2Adaptability or versatility
If wide-field-of-view video and virtual reality head-mounted displays are used, then immersive real-world environment experience is enabled, but bandwidth requirements and data transmission complexity increase
Solution Approach 1:
The system delivers different quality levels of video content to different participants based on their needs and capabilities. Host-side users receive full wide-field-of-view 360-degree video feeds, while remote participants receive targeted video streams focused on their avatar representations and selected regions of interest, optimizing bandwidth utilization while maintaining immersive capabilities.
Solution Approach 2:
The system transmits complete wide-field-of-view video data to the host device for processing, then selectively transmits only the necessary portions (avatars, focused video feeds, orientation data) to remote clients. This partial transmission approach reduces overall data volume while preserving the immersive experience through synthesized visual representations.
3Reliability
If avatars and visual representations are displayed with orientation information, then sense of presence and interaction is enhanced, but processing requirements and computational complexity increase
Solution Approach 1:
The host device acts as an intermediary that receives raw video and orientation data from multiple sources, processes this information into synthesized visual representations (avatars with orientation), and delivers the processed content to remote participants. This centralized processing approach enhances presence through accurate orientation representation while managing computational complexity in a single location rather than distributing it across all devices.
Solution Approach 2:
The system creates visual copies (avatars) of remote participants that include orientation information, allowing host-side users to see represented orientations of remote users without requiring the remote users' actual devices to perform complex rendering. The avatar copies convey presence and orientation data efficiently, reducing the processing burden on individual client devices.
Data Source
AI summary
Embodiments described herein enable a teleconference among host-side user(s) at a host site and remotely located client-side user(s), wherein a host device is located at the host site, and wherein each of the client-side user(s) uses a respective client device to participate in the teleconference. A host site audio-visual feed is received from the host device. A client data feed is received from the client device of each client-side user. Orientation information for the respective client-side user using a client device is also received. Each client device is provided with the host site audio-visual feed or a modified version thereof. The host device is provided with, for each client device, the client data feed and the orientation information of the client-side user that is using the client device. This enables host-side and client-side users to display visual representations of one another with their respective orientations.