Multi-Camera Face Swapping with 3D Geometric Consistency
Solution Overview
Problem
Existing face swapping techniques often produce unrealistic-looking facial changes due to imperfect face geometry and lack of geometric consistency, especially in profile views or extreme poses, as they rely on inaccurate geometric data and do not account for the geometry of faces.
Innovation Solution
A machine learning model trained using multiscopic images to convert facial identities by incorporating 3D geometry, which corrects for discrepancies in facial feature positions and shapes, ensuring geometric consistency in face swapping.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing face swapping techniques use neural network models or CG morphable models, then facial identity can be changed in video frames and images, but the results are unrealistic due to imperfect face geometry and lack of geometric consistency
Solution Approach 1:
The patent transitions from 2D image processing to 3D geometry-based processing by capturing facial images from multiple viewpoints and reconstructing three-dimensional facial geometry. This dimensional elevation allows the system to maintain geometric consistency when swapping facial identities across different poses and angles, resolving the unrealistic artifacts produced by traditional 2D-based methods.
Solution Approach 2:
The patent introduces 3D facial geometry as an intermediary representation between the source and target facial identities. Instead of directly mapping 2D facial features, the system uses 3D geometric models as a mediator to ensure that facial feature positions, shapes, and relationships remain geometrically consistent throughout the identity transformation process.
2Productivity
If face swapping is performed using traditional techniques without 3D geometry, then the process is simpler and faster, but the results show unwanted artifacts and look unrealistic
Solution Approach 1:
The patent performs preliminary 3D geometry reconstruction and facial feature extraction from multiscopic images before the actual face swapping operation. By pre-processing and establishing the 3D geometric framework in advance, the system enables faster real-time identity swapping while maintaining high quality results, as the geometric constraints are already established.
3Manufacturing precision
If multiscopic images are used to capture facial data from multiple viewpoints, then 3D geometry can be derived for better geometric consistency, but the device complexity and data processing requirements increase
Solution Approach 1:
The patent employs a camera rig that serves multiple functions: capturing images from multiple viewpoints for 3D reconstruction, providing stereoscopic vision for depth perception, and enabling facial recognition and analysis. This multi-functional design reduces the need for separate specialized devices while achieving accurate 3D facial geometry extraction.
Data Source
AI summary
One embodiment of the present invention sets forth a technique for performing face swapping. The technique includes converting a first input image that depicts a first facial identity from a first viewpoint at a first time into a first latent representation and converting a second input image that depicts the first facial identity from a second viewpoint at the first time into a second latent representation. The technique also includes generating, via a first machine learning model, a first output image that depicts a second facial identity from the first viewpoint based on the first latent representation. The technique further includes generating, via the first machine learning model, a second output image that depicts the second facial identity from the second viewpoint based on the second latent representation.