3D User Representation With Disparity Correction for Live Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing visual communication techniques fail to provide a realistic three-dimensional representation of a user during live streaming events, lacking enhancements that enhance the viewing experience.
Innovation Solution
A system that combines RGB and depth streams to reconstruct and modify a user's upper body as a 3D representation, using disparity prediction and interpolation techniques to adjust and present a 3D view, potentially on a head-mounted device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If 2D video representation is used for visual communication, then device complexity is reduced, but realism and viewing experience are insufficient
Solution Approach 1:
The patent transforms 2D video representations into 3D representations by introducing depth information through RGB-D cameras and performing 3D reconstruction. This dimensionality change enables realistic 3D avatars with proper spatial geometry, lighting, and occlusion handling, directly resolving the contradiction between realism and complexity by providing a structured approach to 3D transformation.
Solution Approach 2:
The patent creates a 3D copy (avatar) of the user from 2D video input. The system captures 2D video frames, reconstructs 3D geometry, and generates a virtual 3D representation that replicates the user's appearance and movements. This copying approach allows realistic representation while managing complexity through efficient 3D model generation and rendering techniques.
2Manufacturing precision
If depth data is used to reconstruct 3D views, then pixel disparities are reduced, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary 3D reconstruction and depth estimation from RGB-D data before final rendering and disparity removal. By pre-processing the depth information and establishing 3D geometry early in the pipeline, the system reduces computational complexity during real-time rendering, thereby reducing processing time while maintaining precision in pixel positioning.
Solution Approach 2:
The patent replaces traditional mechanical or manual methods of 3D scanning and measurement with optical-based RGB-D sensing and computational photography techniques. This substitution enables rapid acquisition of depth data and 3D geometry, significantly reducing processing time while maintaining high precision in 3D pixel positioning through algorithmic approaches.
3Reliability
If interpolation techniques are applied to remove pixel disparities, then visual appeal is improved, but computational complexity increases
Solution Approach 1:
The patent applies interpolation techniques selectively to specific regions where pixel disparities occur, rather than processing the entire image uniformly. By focusing computational resources on problematic areas such as occlusion boundaries and depth discontinuities, the system improves visual quality while managing computational complexity through targeted rather than exhaustive processing.
Data Source
AI summary
Various implementations disclosed herein include devices, systems, and methods that provide a 3D representation of a user over time during live streaming. For example, a process may include obtaining sensor data depicting two-dimensional (2D) representations of an upper body of a user at multiple points in time. The process may further obtain three-dimensional (3D) information corresponding to portions of the 2D representations and predict disparities in 3D views of the upper body of the user produced using the 2D representations and the 3D information. The disparities are predicted to occur between sets of pixels of the 2D representations. The process may further generate changes to reduce the disparities such that the 3D views of the upper portion of the user with the changes reducing the disparities are presented during a communication session by a receiving device.