Visual-Imagination Model Ensemble for Zero-Shot NLP Bias Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale Pretrained Language Models (PLMs) trained solely on texts suffer from human reporting bias due to the frequency of textual statements not always corresponding to their relative likelihood in the world, limiting their zero-shot capabilities.
Innovation Solution
A method and apparatus that integrate an image conversion model and a language model to create a model ensemble, enabling the processing of both textual and visual inputs to enhance zero-shot natural language understanding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large-scale pretrained language models are trained solely on textual data, then the model size and textual pre-training data can be scaled up to improve zero-shot capabilities, but human reporting bias is inevitably introduced because the frequency of textual statements does not correspond to their relative likelihood in the world
Solution Approach 1:
The patent merges text-based language models with image-based visual information to create a multimodal system. The text-based model processes linguistic input while the image-based model processes visual input, and their predictions are combined through ensemble methods. This combination allows the system to leverage both textual and visual modalities, reducing reliance on biased textual data alone and improving zero-shot capability accuracy while mitigating human reporting bias.
Solution Approach 2:
The patent introduces visual information as an intermediary mediator between the input task and the final prediction. Instead of relying solely on potentially biased textual data, the system uses images as an intermediate representation that can provide more accurate world knowledge. The visual modality serves as a bridge that supplements textual information and corrects biases in the training data.
2Measurement precision
If visual information is integrated to supplement textual data, then zero-shot capability accuracy is improved, but device complexity increases due to the need for multiple models and modalities
Solution Approach 1:
The patent segments the overall NLP task into separate text-based processing and image-based processing components. Each modality is handled by dedicated models that process their respective input types independently. The text-based model handles linguistic understanding while the image-based model handles visual understanding, and their results are combined. This segmentation allows for modular design and independent optimization of each component.
Solution Approach 2:
The patent creates a universal framework that can handle both text-based and image-based inputs through a common ensemble mechanism. The system is designed to be multi-functional, accommodating different input modalities (text-only, image-only, or combined) and different task types (classification, question answering, etc.) through the same architectural approach. This universality reduces the need for task-specific customizations despite the increased complexity.
Data Source
AI summary
A method performed by at least one processor includes receiving a first input stream of a task and a second input stream of a solution. The method further includes selecting the first input stream or the second input stream. The method further includes providing the selected input stream to an image conversion model and a language model. The method further includes creating, based on the selected input stream, a model ensemble of the conversion model and the language model. The method further includes outputting a prediction based on the model ensemble. The method may further include generating an image corresponding to text, converting a textual task into a multimodal task, and solving the multimodal task.