Image-Conditioned Masked Language Modeling for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Collecting structured annotated data for complex visual tasks is difficult and expensive, especially for domains with insufficient data, requiring fine-grained analysis, which hinders the training of accurate and robust neural network models for applications like self-driving cars and autonomous assistants.
Innovation Solution
Training convolutional neural networks using weaker supervision signals, such as noisy data from social media platforms, and employing image conditioned masked language modeling to learn visual embeddings from image-text pairs, where the model predicts masked tokens based on visual information, reducing the need for extensive manual annotations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large-scale manually annotated datasets are used for training deep learning models, then model accuracy and robustness are improved, but the cost and complexity of data collection increase significantly
Solution Approach 1:
The system enables models to learn from unannotated data by performing self-supervised learning through pretext tasks. The model automatically generates its own training signals by predicting masked tokens or solving reconstruction tasks, eliminating the need for manual annotation while still achieving robust representations.
Solution Approach 2:
The patent applies preliminary unsupervised pre-training on abundant unannotated data before fine-tuning on smaller annotated datasets. This preliminary action of learning general representations from raw data without labels reduces the subsequent need for extensive manual annotation while improving final model performance.
2Measurement precision
If fine-grained analysis is performed on input images for complex visual tasks, then task performance is improved, but the amount of structured annotated data required increases
Solution Approach 1:
The patent segments the learning process into two phases: unsupervised pre-training on general visual features from unannotated data, followed by supervised fine-tuning on specific tasks with limited annotations. This segmentation allows the model to learn fine-grained capabilities efficiently without requiring extensive annotated data for each specific task.
Solution Approach 2:
The patent trains a universal visual representation model that can be applied across multiple downstream tasks including image classification, object detection, and segmentation. By learning general fine-grained features from unannotated data through self-supervised tasks, the model achieves multi-task capability without requiring task-specific annotated datasets for each application.
3Reliability
If more competent models are trained for critical jobs, then model reliability is improved, but computational complexity and training costs increase
Solution Approach 1:
The patent employs continuous self-supervised pre-training on large volumes of unannotated data, maintaining continuous learning and representation refinement without interruption. This continuous useful action of learning from abundant data sources improves model competence while distributing computational costs over time rather than requiring intensive short-term training.
Solution Approach 2:
The patent replaces the traditional mechanical approach of manual data annotation with automated self-supervised learning mechanisms. Instead of human annotators manually labeling data, the system uses automated pretext tasks like masked token prediction and image reconstruction to generate training signals, significantly reducing computational and human costs while improving model reliability.
Data Source
AI summary
A method and system pre-trains a convolutional neural network for image recognition based upon masked language modeling by inputting, to the convolutional neural network, an image; outputting, from the convolutional neural network, a visual embedding tensor of visual embedding vectors; tokenizing a caption to create a list of tokens, at least one token having visual correspondence to the image received by the convolutional neural network; randomly selecting one of the tokens in the list of tokens to be masked, the selected token being taken as ground truth; computing, using a language model neural network, hidden representations of the tokens; using the hidden representation of the masked token, as a query vector, to pool the visual embedding vectors in the visual embedding tensor, attentively; predicting the masked token by mapping the pooled visual embedding vectors to the tokens; determining a prediction loss associated with the masked token; and back-propagating the prediction loss to the convolutional neural network to tune parameters thereof.