Template-Based Video Generation for Mobile Face Replacement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current messaging applications lack the ability to perform complex video editing, such as replacing one face with another, requiring sophisticated third-party software and limiting user interaction.
Innovation Solution
A system and method for template-based generation of personalized videos using video configuration data, including frame images, face area parameters, and facial landmark parameters, allowing real-time modification and insertion of a user's face with predefined facial expressions on mobile devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If complex video editing is performed using third-party software, then editing capability is improved, but device complexity and ease of operation deteriorate
Solution Approach 1:
The video editing system is segmented into modular components: template management module, face detection module, face replacement module, and video rendering module. Each module handles a specific aspect of the editing process, allowing complex functionality to be achieved through coordinated simple operations rather than a monolithic complex system
Solution Approach 2:
Pre-defined video templates serve as intermediaries between the user and the complex video editing process. Users simply select a template and provide a face image, while the system automatically handles the complex tasks of face detection, landmark matching, and video composition through the intermediary template framework
2Adaptability or versatility
If complex video editing is performed using third-party software, then editing capability is improved, but ease of operation deteriorates
Solution Approach 1:
Video templates are prepared in advance with pre-defined face regions, landmark positions, and composition parameters. This preliminary action eliminates the need for users to perform complex editing operations during runtime, as the system only needs to substitute the target face into the pre-prepared template structure
Solution Approach 2:
The system automatically performs face detection, landmark identification, and video composition without requiring user intervention for these complex tasks. The automated processing handles the technical complexity while the user only needs to provide simple inputs (face image and template selection)
3Productivity
If face replacement is performed in real-time on mobile devices, then productivity is improved, but use of energy increases
Solution Approach 1:
Video templates are pre-processed and prepared before runtime, with face regions, landmarks, and composition parameters predetermined. This allows real-time execution to focus only on face image substitution and basic composition operations, significantly reducing computational energy requirements compared to performing all processing from scratch
Solution Approach 2:
The system performs only the necessary minimum processing for real-time operation: detecting the source face, matching landmarks with the template, and compositing the image. Full-precision processing is applied only where necessary, rather than uniformly across the entire video processing pipeline
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed are systems and methods for template-based generation of personalized videos. An example method may commerce with receiving a video template including video configuration data (410). The video configuration data (410) may include a sequence (445) of frame images featuring at least one body, a sequence of face area parameters defining positions of a face area in the frame images, and a sequence of skin masks defining positions of a skin area of a part of the at least one body in the frame images. The method may include receiving an image (405) of a source face. Based on the image (405) of the source face, the method may then determine color data associated with the source face. For a frame image of the sequence of frame images, the method may recolor, based on the color data, the skin area of the part of the at least one body in the frame image, and insert the image (405) of the source face into the frame image at a position determined by face area parameters corresponding to the frame image to generate an output frame of an output video.