Foveated 3D Communication Streaming Using Bilateral Attention Exchange
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
3D communication technologies require significant compute resources and bandwidth, leading to issues like latency and unreliable communication due to the large amount of data needed for immersive experiences, which can overwhelm existing infrastructure.
Innovation Solution
Implementing a bilateral exchange of user attention streams to generate foveated communication streams that reduce data transmission by providing high detail only where user attention is focused and lower detail elsewhere, based on shared attention data between communication devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full-detail 3D communication streams are transmitted to provide immersive experiences, then visual quality and immersion are improved, but bandwidth consumption and processing resources increase significantly
Solution Approach 1:
The patent applies local quality by transmitting high-detail 3D data only in regions where the remote user is directing their attention (foveated regions), while transmitting lower-detail or compressed data in peripheral regions. This is achieved by exchanging attention streams between devices to identify regions of interest, then dynamically adjusting the level of detail in the 3D communication stream accordingly, thereby reducing overall data transmission while maintaining visual quality where it matters most.
2Reliability
If high-resolution 3D representations are transmitted continuously, then communication quality is maintained, but latency and network reliability deteriorate
Solution Approach 1:
The patent implements dynamics by continuously adjusting the detail level of transmitted 3D data based on real-time attention information exchanged between devices. Rather than transmitting static full-detail streams, the system dynamically modifies data transmission characteristics according to where the user is looking, which regions require high fidelity, and how attention is shifting over time, thereby reducing latency while maintaining quality where needed.
3Loss of information
If comprehensive 3D data is transmitted to capture all scene details, then information completeness is improved, but processing power and computational resources are overwhelmed
Solution Approach 1:
The patent applies the extraction principle by selectively extracting and transmitting only the most relevant portions of 3D scene data based on attention information. Instead of processing and transmitting all scene details, the system extracts high-detail data specifically from regions where the remote user is directing their attention, while using compressed or lower-fidelity representations for other regions, thereby reducing computational load while preserving important information.
Data Source
AI summary
A first three-dimensional (3D) communication device used by a first user may engage in a 3D communication session with a second 3D communication device used by a second user. For example, the first communication device may: transmit, to the second communication device, a first attention stream indicating where attention of the first user is directed; receive, from the second communication device, a second attention stream indicating where attention of the second user is directed; transmit, to the second communication device and based on the second attention stream, a first foveated image stream including a 3D representation of the first user; receive, from the second communication device, a second foveated image stream that is transmitted based on the first attention stream and includes a 3D representation of the second user; and present the second foveated image stream for viewing by the first user.