Image-Text Pair Dataset Cleaning via Candidate Generation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing large-scale image-text pair datasets are of low quality, containing mismatched image-text content and noise data, which affects the performance of content understanding and content generation models.

Innovation Solution

A method is described for cleaning and re-labeling a low-quality training set to obtain a high-quality image-text pair dataset, by using a content understanding model to generate candidate texts and a content generation model to generate candidate images, and then performing similarity matching to determine a target text for each image-text pair.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If low-quality crawled image-text data is used for training, then the quantity of training data is large, but the model performance deteriorates due to mismatched content and noise

Engineering Contradiction:

Improvequantity of training dataVSAvoidmodel performance

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The patent applies preliminary action by using an initial content understanding model to generate candidate texts for images in the noise-containing sample set before final model training. This pre-processing step creates a cleaner dataset foundation, allowing the model to learn from higher quality associations without the harmful effects of mismatched data, thus resolving the contradiction between data quantity and model performance

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an intermediary mechanism by using candidate text generation and similarity matching as a bridge between the noisy crawled data and the final training set. This intermediary process filters and re-labels data to improve quality while maintaining scale, enabling the system to achieve both large data quantity and high model performance

Inventive Principle:

Principle #24Intermediary (Mediator)

2Manufacturing precision

If manual cleaning of low-quality data is performed, then the quality of training data improves, but the time and cost increase significantly

Engineering Contradiction:

Improvedata qualityVSAvoidcleaning time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent implements self-service by using the content understanding model to automatically generate candidate texts for image re-labeling without human intervention. The system self-corrects its own training data by generating and evaluating multiple candidate descriptions, then selecting the best match through automated similarity comparison, eliminating the need for expensive and time-consuming manual cleaning while maintaining high data quality

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces the mechanical process of manual data cleaning with an automated computational system. Instead of human reviewers manually examining and correcting each image-text pair, the system uses machine learning models to generate candidate texts and automated similarity matching algorithms to select the best matches, substituting human labor with computational processes that are both faster and more scalable

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Measurement precision

If candidate texts are generated and similarity matching is performed, then the accuracy of image-text pairing improves, but the computational complexity increases

Engineering Contradiction:

Improveimage-text matching accuracyVSAvoidcomputational complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent applies partial action by generating only a limited number of candidate texts (e.g., top K candidates) for each image rather than exhaustively generating all possible descriptions. This partial approach achieves sufficient matching accuracy for practical purposes while avoiding the prohibitive computational complexity of exhaustive generation and comparison, finding an optimal balance between precision and computational burden

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20250157197A1Methods and apparatuses for training content understanding model and content generation model

Publication Date: 2025.05.15 ALIPAY (HANGZHOU) INFORMATION TECH CO LTD

AI summary

Separately training a content understanding model and a content generation model with an image-text pair formed by an image and a text in a target training set. Sample processing is performed on a noise-containing sample set, including: inputting a first image in any first image-text pair in the noise-containing sample set into the content understanding model to obtain several candidate texts; separately inputting a first text in the first image-text pair and the several candidate texts into the content generation model to obtain multiple candidate images; performing similarity matching between the multiple candidate images and the first image to determine a target text; and continuing to train the content understanding model and the content generation model by forming a second image-text pair by using the first image and the target text and adding the second image-text pair to the target training set.