3D Face Parameter Compression for Ultra-Low-Bitrate Talking Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video compression technologies do not effectively incorporate semantic information related to human motion, particularly in ultra-low bit-rate talking face communication, leading to inefficiencies in encoding and decoding processes.
Innovation Solution
The method and apparatus utilize a 3-dimensional (3D) face representation to generate and reconstruct face parameters for key and inter frames, enhancing the encoding and decoding processes by incorporating semantic information related to human motion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If traditional video compression technologies are used, then storage memory and transmission bandwidth are reduced, but encoding and decoding efficiency deteriorates in ultra-low bit-rate talking face communication
Solution Approach 1:
The patent segments the video encoding process by separating key frames (encoded using traditional compression) from inter frames (encoded using 3D face representation). This segmentation allows efficient handling of talking face videos by focusing computational resources on extracting and encoding only the essential 3D face parameters rather than processing entire frame sequences, thereby improving encoding efficiency while maintaining low bit-rate transmission
Solution Approach 2:
The patent changes the encoding parameters from traditional pixel-based 2D frame data to 3D face representation parameters (such as 3D mesh vertices, texture coordinates, and pose parameters). This parameter transformation enables more efficient representation of talking face videos at ultra-low bit-rates by capturing only the essential facial geometry and motion information, thus improving both compression efficiency and decoding performance
2Productivity
If 3D face representation is used to enhance encoding and decoding, then video compression efficiency improves, but device complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-processing video frames to extract 3D face parameters before compression. The system performs 3D face model fitting and parameter extraction in advance, transforming the video data into a compact 3D representation format. This preliminary transformation simplifies the subsequent compression and decoding processes, as the decoder only needs to reconstruct frames from the pre-extracted 3D parameters rather than performing complex 3D reconstruction in real-time
Solution Approach 2:
The patent extracts only the essential 3D face representation parameters from full video frames, separating the critical facial geometry and motion information from redundant visual data. By extracting and encoding only these key parameters (such as 3D mesh control points and pose angles) rather than entire frame sequences, the system reduces computational complexity while maintaining high compression efficiency for talking face videos
Data Source
AI summary
Methods and apparatuses are provided for processing video data. An exemplary method includes: decompressing a compressed frame to generate a key frame representing a face; generating, for the key frame, a first set of parameters associated with a 3-dimensional (3D) face representation of the face; reconstructing, for each of one or more inter frames, a second set of parameters associated with a 3D face representation of the face according to compressed inter-predicted residuals of the second set of parameters; and generating a video comprising the face based on the key frame, the first set of parameters, and the second set of parameters.