Multimodal Transformer for Unified Visual Attention Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine-learned models focus on either implicit or explicit human behavior in isolation, lacking diverse generalization capabilities and requiring multiple systems for different tasks.
Innovation Solution
A unified machine-learned multimodal model that integrates both implicit and explicit human behavior predictions, using a multimodal transformer and distinct prediction heads to generate heatmaps, scanpaths, and ratings across diverse visual content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate models are used for implicit and explicit behavior prediction, then each model can be optimized for its specific task, but the overall system complexity increases and resource requirements multiply
Solution Approach 1:
The patent merges separate models for implicit and explicit behavior prediction into a single unified multimodal model. The model integrates multiple prediction heads (heatmap predictor, sequence predictor, rating predictor) within one architecture, sharing common embedding layers and transformer encoders. This consolidation maintains prediction accuracy while reducing system complexity and resource requirements compared to maintaining separate specialized models.
Solution Approach 2:
The unified multimodal model is designed with multi-functionality to perform multiple prediction tasks simultaneously. The single model can generate implicit behavior predictions (heatmaps, scanpaths) and explicit behavior predictions (ratings) by processing the same input through different prediction heads, making it a universal solution that replaces multiple specialized models.
2Reliability
If task-specific models are developed, then performance on specific tasks is optimized, but generalization capability across diverse tasks is limited
Solution Approach 1:
The model achieves universality by implementing a single multimodal architecture that can handle diverse prediction tasks across different domains. The shared embedding layers and transformer encoder learn general representations from diverse inputs (images, text, audio), enabling the model to generalize to unseen tasks while maintaining high performance on trained tasks through the appropriate prediction heads.
3Reliability
If multiple separate systems are deployed for different behavior prediction tasks, then each task receives dedicated computational resources, but total resource consumption increases
Solution Approach 1:
The patent combines multiple prediction systems into one unified model that shares computational resources. The shared embedding layers and transformer encoder are processed once for all prediction heads, avoiding redundant computations. This merging approach maintains prediction quality through multiple specialized heads while significantly reducing total computational resource consumption compared to running separate dedicated systems.
Data Source
AI summary
Aspects of the disclosed technology include computer-implemented systems and methods for machine-learned multimodal models. A machine-learned multimodal model includes one or more embedding layers configured to generate one or more image tokens and one or more text tokens in response to the imagery and the text, a transformer encoder configured to receive the one or more image tokens and the one or more text tokens and generate one or more fused image tokens and one or more fused text tokens, a heatmap predictor configured to obtain the one or more fused image tokens and generate at least one image heatmap, and a sequence predictor configured to obtain the one or more fused image tokens and the one or more fused text tokens and generate a predicted sequence associated with the image.