Face Restoration Coding for Low-Bandwidth Video Conferencing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video conferencing technologies using AI-based face re-enactment methods suffer from lack of compressibility, accuracy, and sensitivity to occlusion and large motion, leading to artifacts and inadequate fidelity in reconstructed faces.
Innovation Solution
A method involving face detection, facial landmark extraction, and extended face area feature compression, followed by spatial-temporal down-sampling and efficient transmission of landmark and EFA features, with reconstruction at the decoder side using codebooks and video compression techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If AI-based face re-enactment methods are used to reduce transmission bandwidth, then bandwidth requirements are reduced, but face reconstruction fidelity deteriorates and artifacts are generated
Solution Approach 1:
The patent segments the face into multiple regions (face region and extended face area) and transmits different levels of detail for each. The face region uses landmark features for compression, while the extended face area uses additional texture information, allowing selective quality allocation based on importance and reducing overall bandwidth while maintaining fidelity.
Solution Approach 2:
The patent applies different quality levels to different regions of the face. High-quality texture information is transmitted for the extended face area (eyes, eyebrows, nose) where detail is critical, while the face region uses compressed landmark data, optimizing the balance between bandwidth and reconstruction quality.
2Loss of energy
If face re-enactment framework is used to compress video data, then transmission bit consumption is reduced, but compressibility and accuracy of neural network processing deteriorate
Solution Approach 1:
The patent introduces an intermediary representation layer using facial landmark features that bridge the original video data and the compressed transmission. These landmarks serve as a mediator that captures essential facial information while enabling efficient compression, and can be accurately recovered and processed by neural networks for reconstruction.
3Loss of energy
If 2D or 3D facial landmark features are extracted and transmitted, then bandwidth consumption is reduced, but sensitivity to occlusion and large motion increases causing artifacts
Solution Approach 1:
The patent performs preliminary action by extracting and transmitting both landmark features and extended face area texture information before the actual video conferencing. This preliminary extraction of multiple feature types ensures that sufficient information is available at the decoder to handle occlusion and large motion scenarios, reducing sensitivity to these challenging conditions.
Solution Approach 2:
The patent changes the parameters of facial representation by including not only traditional 2D/3D landmarks but also extended face area texture information and pose data. This expanded parameter set provides more robust information for handling occlusion and motion, improving reconstruction accuracy under challenging conditions.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
There is included a method and apparatus comprising computer code configured to cause a processor or processors to perform obtaining video data, detecting at least one face from at least one frame of the video data, determining a set of facial landmark features of the at least one face from the at least one frame of the video data, and coding the video data at least partly by a neural network based on the determined set of facial landmark features.