Virtual Camera Viewpoint Shift for Natural Video Eye Contact
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video conferencing systems with a single camera mounted off-center create an unnatural experience by forcing direct eye contact without allowing gaze deviations, and 3D modeling for reorienting the head is computationally intensive and power-consuming.
Innovation Solution
Employing motion estimation and interpolation techniques to synthesize a virtual camera view using 2D processing, estimating motion vectors between reference and active images to shift the camera viewpoint, allowing natural gaze deviations while using a single camera.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If 3D modeling and manipulation is used to reorient the head toward the camera, then eye contact is improved, but computational intensity and power consumption increase
Solution Approach 1:
The patent creates a virtual copy of the camera at the screen center position. Instead of moving or reorienting the physical camera or performing complex 3D head reorientation, the system synthesizes a virtual camera view that appears to originate from the center of the screen, thereby improving eye contact while avoiding the computational costs of 3D modeling
Solution Approach 2:
The patent replaces the mechanical/physical approach of 3D head modeling and reorientation with a 2D image processing approach. By using 2D frame interpolation and motion compensation on the captured video frames, the system achieves the desired eye contact effect without the computational intensity of 3D physics-based modeling
2Reliability
If eye gaze is modified without reorienting the head, then direct eye contact is achieved, but natural gaze deviations are lost
Solution Approach 1:
The patent dynamically adjusts the virtual camera viewpoint based on the user's actual gaze direction. Rather than forcing a fixed direct eye contact, the system allows the synthesized view to adapt to natural gaze deviations, maintaining eye contact quality when the user looks at the screen while preserving natural behavior when the user looks away
Solution Approach 2:
The patent introduces a virtual camera as an intermediary between the physical off-center camera and the user's eyes. This virtual camera, positioned at the screen center, mediates the visual relationship by synthesizing frames that appear to come from the correct position, thereby restoring natural eye contact without constraining natural gaze movements
3Reliability
If a physical camera is positioned at the center of the viewing screen, then natural eye contact is achieved, but device design and camera mounting become more complex
Solution Approach 1:
The patent creates a virtual copy of the camera positioned at the screen center, eliminating the need to physically relocate the camera. This virtual camera copy provides the benefits of center-positioned imaging without the mechanical complexity of repositioning hardware components
Solution Approach 2:
The patent replaces the mechanical solution of physically moving the camera to the screen center with a computational approach. By using 2D image processing, motion estimation, and frame interpolation, the system achieves center-positioned perspective without any physical camera repositioning, thereby maintaining simple device design
Data Source
AI summary
The present invention provides systems and methods for motion compensated single camera viewpoint shifting. One or more reference views of a user looking at one of a camera and a screen are captured. Further, for a current active screen view, a virtual camera view is synthesized by motion compensated interpolation between the one or more reference views and the current active screen view. The virtual camera view may comprise a virtual image of the user looking at the screen. The reference views may comprise at least one of: one or more training camera views of the user looking into the camera, each captured at a different orientation of the user relative to the camera; and one or more training screen views of the user looking at the screen, each captured at a different orientation of the user relative to the screen.