3D Holographic Calling Pipeline for Realistic Spatial Interaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video conferencing systems fail to accurately depict body language and allow for relational movements between participants, introducing a fixed point of view and an intrusive layer of technology that diminishes the in-person communication experience.
Innovation Solution
A holographic calling system that captures and encodes holographic data at the sender-side, performing stages such as depth image densification, body modeling, facial reconstruction, and torso disocclusion, and decodes it as a 3D representation at the receiver-side, allowing users to interact with a hologram in 3D space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If video conferencing is used to connect participants, then communication capability is improved, but the realism of in-person interaction deteriorates due to 2D representation and fixed point of view
Solution Approach 1:
The patent transitions from 2D video representation to 3D holographic representation, enabling participants to be viewed from multiple angles and positions. This dimensional change preserves body language and contextual information by allowing relational movements between participants, making the communication experience more similar to in-person interaction.
2Ease of operation
If video calling on flat panel display is used, then communication accessibility is improved, but the perception of in-person communication deteriorates due to intrusive technology layer
Solution Approach 1:
The patent replaces flat 2D panel displays with volumetric 3D holographic displays that render participants as three-dimensional objects in space. This eliminates the intrusive flat screen barrier, allowing users to perceive depth, volume, and spatial relationships, thereby restoring the perception of in-person communication while maintaining accessibility through various display interfaces.
3Device complexity
If fixed camera point of view is used in video calling, then system simplicity is improved, but the ability to perform relational movements deteriorates
Solution Approach 1:
The patent implements a multi-functional system where the holographic calling platform supports various viewing angles, positions, and orientations through 3D spatial representation. Participants can move freely in virtual space, perform gestures, and interact relationally, all while maintaining a unified holographic framework that handles multiple functions (viewing, interaction, spatial positioning) within a single system architecture.
Data Source
AI summary
A holographic calling system can capture and encode holographic data at a sender-side of a holographic calling pipeline and decode and present the holographic data as a 3D representation of a sender at a receiver-side of the holographic calling pipeline. The holographic calling pipeline can include stages to capture audio, color images, and depth images; densify the depth images to have a depth value for each pixel while generating parts masks and a body model; use the masks to segment the images into parts needed for hologram generation; convert depth images into a 3D mesh; paint the 3D mesh with color data; perform torso disocclusion; perform face reconstruction; and perform audio synchronization. In various implementations, different of these stages can be performed sender-side or receiver side. The holographic calling pipeline also includes sender-side compression, transmission over a communication channel, and receiver-side decompression and hologram output.