Masked Autoencoder Inpainting with Dual Pixel-Feature Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing self-supervised representation learning techniques in computer vision, such as Masked Image Modeling, are limited by pixel-level prediction targets, leading to inefficient training and model memorization of target-specific information, which hinders effective inpainting of masked images.
Innovation Solution
A method involving a primary encoding process that encodes visible image patches, with separate pixel regression and feature prediction decoders, using a momentum encoder for dynamic targets and feature injection to reduce target-specific learning pressure, focusing on structural feature modeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pixel-level prediction targets are used in masked image modeling, then the model can perform inpainting tasks, but the model memorizes target-specific information and training efficiency decreases
Solution Approach 1:
The patent segments the prediction task into two independent branches: pixel regression branch for pixel-level prediction and feature prediction branch for semantic feature prediction. This segmentation allows each branch to specialize in different aspects, preventing the model from memorizing target-specific information while maintaining inpainting accuracy through the feature prediction branch that focuses on semantic understanding.
Solution Approach 2:
The patent extracts the feature prediction function from the pixel-level prediction process by introducing a separate feature prediction branch with its own decoder. This extraction allows the model to separate semantic feature learning from pixel-level reconstruction, thereby improving training efficiency by reducing the memorization burden on the encoder while maintaining reliable inpainting through the specialized feature prediction pathway.
2Manufacturing precision
If pixel-level prediction targets are used, then the model learns to reconstruct images, but it wastes training effort on target-specific information
Solution Approach 1:
The patent divides the training objective into two independent loss functions: pixel regression loss for reconstruction accuracy and feature prediction loss for semantic feature learning. This segmentation allows the model to allocate training effort efficiently by processing both tasks simultaneously through separate branches, avoiding wasted effort on target-specific information while maintaining image reconstruction accuracy through the pixel regression branch.
Solution Approach 2:
The encoder is designed to serve multiple functions through the dual-branch architecture: it encodes visual information for both pixel-level reconstruction (via the pixel regression branch) and semantic feature prediction (via the feature prediction branch). This multi-functionality allows the model to learn universal representations that improve both image reconstruction accuracy and training efficiency by leveraging shared encoder computations.
3Device complexity
If a single decoding process is used for both pixel and feature prediction, then the model structure is simple, but the model cannot effectively learn semantic features
Solution Approach 1:
The patent segments the decoding process into two separate decoders: a pixel regression decoder for pixel-level prediction and a feature prediction decoder for semantic feature extraction. This segmentation enables each decoder to be optimized for its specific task, with the feature prediction decoder dedicated to learning semantic features through the feature prediction branch, thereby improving semantic feature learning precision while maintaining manageable model complexity through modular architecture.
Solution Approach 2:
The patent applies local quality by assigning different functional characteristics to different parts of the model: the pixel regression branch focuses on pixel-level reconstruction with appropriate loss functions and activation functions, while the feature prediction branch focuses on semantic feature extraction with specialized decoding mechanisms. This local differentiation allows each component to optimize its specific function, improving semantic feature learning while keeping the overall model structure organized and manageable.
Data Source
AI summary
The disclosure herein describes training an encoder network to inpaint images with masked portions. A primary encoding process is used to encode a visible portion of a masked input image into encoded token data. The encoded token data is then decoded into both pixel regression output and feature prediction output, wherein both outputs include inpainted image data associated with the masked portion of the masked input image. A pixel regression loss is determined using the pixel regression output and pixel data of an unmasked version of the masked input image. A feature prediction loss is determined using the feature prediction output and ground truth encoding output of the unmasked version of the masked input image. The primary encoding process is then trained using the pixel regression loss and the feature prediction loss, whereby the primary encoding process is trained to encode structural features of input images into encoded token data.