Masked Partial Convolution for Irregular-Hole Image In-Painting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image in-painting techniques struggle to effectively fill irregular holes in images using deep learning approaches, often resulting in artifacts like lack of texture and artificial edge responses, and are limited by fixed initial pixel values and reliance on image statistics.
Innovation Solution
A deep learning neural network performs partial convolution operations using a mask to identify valid and invalid pixels, updating the mask after each operation to fill irregular holes with synthesized pixel data, incorporating visual semantics and reducing artifacts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep learning neural networks are used to perform image in-painting, then visual semantics can be incorporated into the solution, but artifacts such as lack of texture and artificial edge responses manifest in the hole regions
Solution Approach 1:
The patent applies partial convolution operations that selectively process only valid (non-hole) pixels while excluding invalid pixels from the convolution computation. This partial action prevents the harmful propagation of hole region information into synthesized pixels, thereby reducing artifacts like artificial edge responses and texture deficiencies while maintaining the benefits of deep learning-based visual semantics.
2Device complexity
If fixed initial values are used for pixels in the hole, then the in-painting process can be simplified, but the results are skewed and produce unrealistic synthesized pixel data
Solution Approach 1:
The patent replaces fixed initial values with dynamic initialization that adapts to each pixel's local context. Invalid pixels are initialized using a weighted combination of neighboring valid pixels, where weights are determined by spatial distance and mask values. This dynamic approach maintains process simplicity while significantly improving synthesized pixel data quality by avoiding the skewing effect of fixed initial values.
3Productivity
If convolution operations are performed including invalid pixels in holes, then computation is simplified, but invalid pixel data propagates to synthesized pixels creating artifacts
Solution Approach 1:
The patent segments the image pixels into valid and invalid categories using a binary mask, and applies different convolution operations to each segment. Valid pixels undergo standard convolution operations, while invalid pixels undergo partial convolution that excludes them from contributing to synthesized pixel values. This segmentation prevents invalid pixel data propagation while maintaining computational efficiency through vectorized operations.
4Device complexity
If regular rectangular holes are processed, then the in-painting algorithm can be simplified, but irregularly shaped holes cannot be effectively handled
Solution Approach 1:
The patent applies local quality by using a spatial mask that varies across different regions of the image to represent irregular hole shapes. Each pixel's processing is adapted to its local context - pixels near hole boundaries use different convolution kernels and weighting schemes compared to pixels in the middle of hole regions. This enables effective handling of irregularly shaped holes while maintaining algorithmic simplicity through localized adaptations.
Data Source
AI summary
A neural network architecture is disclosed for performing image in-painting using partial convolution operations. The neural network processes an image and a corresponding mask that identifies holes in the image utilizing partial convolution operations, where the mask is used by the partial convolution operation to zero out coefficients of the convolution kernel corresponding to invalid pixel data for the holes. The mask is updated after each partial convolution operation is performed in an encoder section of the neural network. In one embodiment, the neural network is implemented using an encoder-decoder framework with skip links to forward representations of the features at different sections of the encoder to corresponding sections of the decoder.