Image Resynthesis via Forward Warping and Gap Discriminators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image resynthesis methods, particularly those using backward warping, face challenges in accurately predicting new views of a person or object due to misalignment between the source image and the warping field, leading to suboptimal quality and complexity in scene modeling.
Innovation Solution
The proposed method employs a forward warping module to predict corresponding positions in the target image for each source image pixel, aligned with the source image, and a gap filling module to fill in gaps resulting from forward warping, using a deep convolutional neural network architecture and a gap discriminator for improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If backward warping is used to predict pixel locations in source image for each target pixel, then the warping process can be implemented, but misalignment between source image and warping field occurs leading to reduced accuracy
Solution Approach 1:
The patent inverts the conventional backward warping approach by using forward warping. Instead of predicting where source pixels map to in the target image, the method predicts where target pixels map to in the source image. This inversion aligns the warping field with the source image coordinate system, eliminating misalignment issues and improving prediction accuracy while simplifying the scene modeling process.
2Measurement precision
If forward warping is used to predict corresponding positions aligned with source image, then accuracy is improved, but gaps are created in the target image requiring additional processing
Solution Approach 1:
The patent extracts and separately handles the gap filling operation from the main warping process. By using forward warping, gaps are systematically created in predictable locations. The method then extracts these gap regions and fills them using specialized techniques, separating the warping and gap filling operations to improve overall efficiency while maintaining high accuracy in the alignment process.
3Reliability
If deep convolutional networks are used for image resynthesis, then quality of synthesized images is improved, but computational resources and training complexity increase
Solution Approach 1:
The patent segments the image resynthesis task into distinct functional modules: forward warping module, gap filling module, and refinement module. Each module is implemented as a separate deep convolutional network trained for its specific function. This segmentation allows for more efficient training and inference compared to a single monolithic network, reducing overall computational resource requirements while maintaining high image quality through specialized processing at each stage.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The invention relates to image processing and, in particular, to image resynthesis for synthesizing new views of a person or an object based on an input image, to resolve tasks such as predicting views of a person or an object from new viewpoints and in new poses. Technical result consists in improved accuracy of image resynthesis based on at least one input image. An image resynthesis system, a system for training a gap filling module to be used in the image resynthesis system, an image resynthesis method, a computer program product and a computer-readable medium are provided. The image resynthesis system comprises a source image input module, a forward warping module configured to predict, for each source image pixel, a corresponding position in a target image, the forward warping module being configured to predict a forward warping field which is aligned with the source image, and a gap filling module configured to fill in the gaps resulting from the application of the forward warping module. The image resynthesis method comprises the steps of: inputting a source image, predicting, for each source image pixel, a corresponding position in a target image, wherein a forward warping field which is aligned with the source image is predicted, predicting a binary mask of gaps which result from the forward warping, filling in the gaps based on said binary mask of gaps by generating a texture image by means of predicting a pair of coordinates in the source image for each pixel in the texture image, and mapping the whole texture back to a new pose using backward warping.