Virtual Object Figure Synthesis with 3D Face Key Point Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for synthesizing virtual object images fail to accurately reflect human speech by losing information from the original mouth image during fusion, resulting in a lack of correspondence between the mouth shape in the virtual object and the original image.
Innovation Solution
A method that processes face key points to generate 3D face position and posture information for the virtual object, and vertex information for original face images, allowing for the synthesis of a target face image that aligns with the virtual object's posture and position, thereby achieving natural image fusion that retains the original mouth shape.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If coordinate relationships of key points are extracted and mapped from mouth image to virtual object image, then the virtual object can reflect human speaking mouth shape, but the original mouth image information is completely lost and cannot achieve true image fusion
Solution Approach 1:
The patent creates a digital twin (target face image) that copies the essential features of the original face image including mouth shape information, rather than simply mapping coordinates. This digital twin retains the original mouth image characteristics while being adaptable to the virtual object's coordinate system, thus preventing information loss during the fusion process
Solution Approach 2:
The patent transforms the fusion approach by changing from direct coordinate mapping to a parameter-based transformation. It extracts key point parameters from both the virtual object and original face image, processes them through algorithms to generate position and posture information, and uses these parameters to create a target face image that preserves original mouth shape characteristics while adapting to the virtual object's spatial parameters
2Adaptability or versatility
If coordinate relationships of key points are mapped from mouth image to virtual object image, then the virtual object can simulate human speaking, but the mouth shape in virtual object image and original mouth image do not correspond to the same expression speech in the same coordinate system
Solution Approach 1:
The patent resolves the coordinate system mismatch by introducing a third dimension. It extracts three-dimensional position information and posture parameters from key points, creating a 3D spatial understanding of both the virtual object and original face image. This dimensional transformation allows accurate correspondence between mouth shapes across different coordinate systems by establishing a common 3D reference framework
Solution Approach 2:
The patent performs preliminary processing of key point extraction and 3D position/posture parameter generation before the actual image fusion. By pre-processing both the virtual object's face key points and the original face image's key points to generate their respective position and posture information in advance, it ensures that both images are transformed into the same coordinate reference system before fusion, guaranteeing precise correspondence
Data Source
AI summary
A method for synthesizing a figure of a virtual object includes: obtaining a figure image of the virtual object, and original face images corresponding to a speech segment; extracting a first face key point of the face of the virtual object, and a second face key point of each of the original face images; processing the first face key point to generate a position and posture information of a first three-dimensional 3D face; processing each second face key point to generate vertex information of a second 3D face; generating a target face image corresponding to each original face image based on the position and the posture information of the first 3D face and the vertex information of each second 3D face; and synthesizing a speaking figure segment of the virtual object, corresponding to the speech segment, based on the figure image of the virtual object and each target face image.