Visual Commonsense Reasoning via VLM-LLM Segmentation

The patent segments the VCR task into two distinct sub-tasks: visual commonsense understanding (VCU) and visual commonsense inference (VCI). VCU is handled by pre-trained VLMs that perform visual recognition and understanding, while VCI is handled by LLMs that perform reasoning and inference. This segmentation allows each component to be optimized independently, with VLMs pre-trained on general visual data and LLMs pre-trained on general language data, eliminating the need for supervised fine-tuning on each specific dataset while maintaining both accuracy and generalization capability.

The patent employs pre-trained VLMs and LLMs that have been trained on universal datasets (ImageNet for VLMs, general text corpora for LLMs) rather than dataset-specific models. These universal models can handle multiple visual commonsense reasoning tasks across different datasets without requiring retraining, achieving both high performance on individual datasets and broad generalization across diverse visual reasoning problems.

The patent divides the VCR functionality between two specialized models: pre-trained VLMs for visual recognition and understanding (VCU), and pre-trained LLMs for commonsense inference (VCI). The VLMs leverage their visual encoding capabilities to accurately perceive and describe image content, while the LLMs leverage their language reasoning capabilities to draw commonsense conclusions from the visual descriptions. This segmentation allows each model to excel at its designated function without compromising the other.

The patent introduces an intermediary mechanism where VLMs generate textual descriptions or representations of visual content that serve as input to LLMs for reasoning. This intermediary textual representation layer allows the visual accuracy of VLMs to be combined with the reasoning capability of LLMs, transforming the limitation of VLMs lacking reasoning into a collaborative system where visual understanding feeds into textual reasoning.

The patent assigns visual understanding responsibilities to pre-trained VLMs while assigning reasoning responsibilities to pre-trained LLMs. The VLMs process images and generate textual representations that capture visual semantics, which then serve as input to LLMs for commonsense inference. This segmentation ensures that visual understanding is performed by models optimized for vision while reasoning is performed by models optimized for language, with each component performing its strength.

The patent uses textual descriptions generated by VLMs as an intermediary that bridges visual understanding and reasoning capabilities. The VLMs translate visual information into text that LLMs can process with their reasoning capabilities. This intermediary textual layer enables LLMs to perform reasoning without needing direct visual processing, while still maintaining accurate visual understanding through the VLMs' descriptions.

The patent segments the VCR task into VCU and VCI sub-tasks that can be handled by different pre-trained models. For VCU, pre-trained VLMs like BLIP-2 are used which have been trained on general visual datasets. For VCI, pre-trained LLMs are used which have been trained on general language data. This segmentation allows the system to achieve strong generalization across different visual commonsense reasoning datasets without requiring task-specific fine-tuning, as each sub-task is handled by a model pre-trained on general data relevant to that function.

The patent leverages the universal capabilities of pre-trained VLMs and LLMs that have been trained on large-scale general datasets. The VLMs are trained on general image datasets like ImageNet, and LLMs are trained on general text corpora, giving them universal capabilities that transfer to multiple visual commonsense reasoning datasets. This universality allows the system to maintain high performance across different tasks and datasets without requiring fine-tuning, as the pre-trained models generalize well to unseen domains.