AI Visual Question Answering Commonsense Reasoning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Visual question answering (VQA) systems face challenges in incorporating nuanced commonsense reasoning, leading to inaccurate or nonsensical answers due to a lack of contextual understanding and explicit knowledge, which affects their reliability and real-world applicability.
Innovation Solution
A method and device for commonsense reasoning in AI, involving sequence-to-sequence answer prompt generation, commonsense scoring, and rescoring, using a pre-trained mask language model and a commonsense knowledge graph to filter and normalize answers, improving the selection of plausible responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If VQA models consider all possible answers without filtering, then they may not miss the correct answer, but they waste time and resources considering nonsensical possibilities
Solution Approach 1:
The system performs preliminary filtering of answer candidates using commonsense knowledge before full evaluation. By pre-identifying and eliminating obviously incorrect answers based on world knowledge, the system reduces the search space and avoids wasting computational resources on nonsensical possibilities while ensuring the correct answer is not prematurely discarded.
Solution Approach 2:
The patent introduces an intermediary component (commonsense reasoning module) that acts as a mediator between the question and the answer evaluation process. This intermediary uses world knowledge to assess the plausibility of answer candidates, providing a preliminary screening layer that guides the subsequent detailed analysis by the VQA model.
2Reliability
If VQA systems embed broad world knowledge, then they improve contextual understanding, but the system complexity increases
Solution Approach 1:
The system segments the knowledge processing into distinct modules: a commonsense reasoning module that handles world knowledge and plausibility assessment, and a VQA model that focuses on visual and question analysis. This segmentation allows broad world knowledge to be embedded in a dedicated component without overwhelming the entire system, making the complexity manageable and targeted.
Solution Approach 2:
The patent introduces an intermediary component (commonsense reasoning module) that acts as a mediator between the question and the answer evaluation process. This intermediary uses world knowledge to assess the plausibility of answer candidates, providing a preliminary screening layer that guides the subsequent detailed analysis by the VQA model.
3Measurement precision
If VQA models use extensive computational resources for analysis, then they improve answer accuracy, but resource utilization increases
Solution Approach 1:
The system applies partial action by performing commonsense filtering on all answer candidates before applying the full computational power of the VQA model. This preliminary filtering step uses less computationally intensive methods to eliminate obviously incorrect answers, allowing the resource-intensive VQA model to focus only on plausible candidates, thereby reducing overall resource utilization while maintaining accuracy.
4Loss of information
If training data does not explicitly mention obvious knowledge, then VQA systems miss implicit information, but adding explicit training data increases data requirements
Solution Approach 1:
The patent introduces an intermediary component (commonsense reasoning module) that acts as a mediator between the question and the answer evaluation process. This intermediary uses world knowledge to assess the plausibility of answer candidates, providing a preliminary screening layer that guides the subsequent detailed analysis by the VQA model.
Solution Approach 2:
The system uses pre-existing commonsense knowledge bases and world knowledge resources that are independently available outside the training data. Rather than requiring all implicit knowledge to be explicitly present in training data, the system serves itself by accessing external commonsense resources to fill gaps in implicit information.
Data Source
AI summary
A method for controlling an artificial intelligence (AI) device can include receiving, via a processor in the AI device, an input image and a query related to the input image, generating, via the processor, an answer prompt template based on the query, the answer prompt template including a sentence containing a mask token located at a position corresponding to an answer within the sentence, and combining the query and the answer prompt template to generate a string of text including the mask token. Also, the method can further include inputting the string of text to a pre-trained mask language module (MLM) and generating a plurality of scores respectfully corresponding to a plurality of answers, each of the plurality of answers being a candidate for replacing the mask token, determining a selected answer among the plurality of answers based on the plurality of scores, and outputting the selected answer.