Meta-Personalized Vision-Language Model Adaptation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Pre-trained Vision-Language Models (VLMs) struggle to reason about new personal concepts, as they rely on fixed vocabularies that cannot understand personal search terms not present in their training data, leading to challenges in personalizing search queries and overfitting issues when learning from few examples.

Innovation Solution

Implementing a meta-personalization approach that combines meta-learning and test-time adaptation techniques to expand the input vocabulary of pre-trained VLMs, allowing them to learn global category features and adapt to personal instances with few examples, using a mining system to automatically identify personal instances in videos without human annotations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a pre-trained VLM uses a fixed vocabulary, then the model structure remains simple and computationally efficient, but the model cannot understand personal search terms not present in training data

Engineering Contradiction:

Improveability to understand personal search termsVSAvoidvocabulary expansion complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system performs preliminary action by automatically mining personal instances from unannotated videos before the personalization process. The mining system extracts personal instances using pattern recognition and video analysis, preparing the data in advance so that the VLM can be personalized without manual annotation effort during deployment.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces an intermediary component - a personal instance miner - that bridges the gap between unannotated video data and the VLM personalization process. This miner automatically identifies and extracts personal instances, serving as a mediator that enables personalization without requiring direct human annotation input.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Use of energy by moving object

If the VLM is personalized with few personal instances, then the computational resources required are reduced, but the model suffers from overfitting and poor generalization

Engineering Contradiction:

Improvecomputational resource requirementsVSAvoidmodel generalization performance

Core Design Contradiction:

Use of energy by moving objectVSReliability

Solution Approach 1:

The system performs preliminary action by automatically mining and curating personal instances from videos before personalization. This pre-processing step ensures that even with few examples, the most relevant and representative instances are selected, improving generalization without requiring large datasets or excessive computational resources.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system extracts only the most relevant personal instances from video data, taking out the essential features needed for personalization while discarding redundant information. This extraction approach enables effective personalization with few examples, reducing computational requirements while maintaining reliability.

Inventive Principle:

Principle #2Taking out (Extraction)

3Measurement precision

If manual annotation is used to identify personal instances, then the accuracy of personal instance identification is high, but the time and labor requirements increase significantly

Engineering Contradiction:

Improvepersonal instance identification accuracyVSAvoidannotation time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system implements self-service by enabling the mining system to automatically identify personal instances without human annotation. The miner uses pattern recognition, video analysis, and metadata processing to autonomously extract personal instances, eliminating the need for manual annotation while maintaining practical accuracy levels.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system replaces the mechanical process of manual annotation with an automated computational mining system. The miner uses algorithmic approaches including pattern matching, video frame analysis, and metadata processing to substitute human annotation efforts, significantly reducing time and labor requirements.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20240419726A1Learning to Personalize Vision-Language Models through Meta-Personalization

Publication Date: 2024.12.19 ADOBE INC

AI summary

Techniques for learning to personalize vision-language models through meta-personalization are described. In one embodiment, one or more processing devices lock a pre-trained vision-language model (VLM) during a training phase. The processing devices train the pre-trained VLM to augment a text encoder of the pre-trained VLM with a set of general named video instances to form a meta-personalized VLM, the meta-personalized VLM to include global category features. The processing devices test the meta-personalized VLM to adapt the text encoder with a set of personal named video instances to form a personal VLM, the personal VLM comprising the global category features personalized with a set of personal instance weights to form a personal instance token associated with the user. Other embodiments are described and claimed.