Vision-Language Action Recognition With Taxonomy-Guided Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current vision-language (VL) models for action recognition require significant computational resources due to large-scale training datasets and model sizes, which is inefficient and costly.
Innovation Solution
A method for generating a domain-specific training dataset tailored to human activities using a large language model (LLM) to create a taxonomy of human actions, retrieve relevant image-text pairs, and enhance captions, enabling a vision-language model to perform action recognition efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large-scale training datasets and model sizes are used for vision-language models, then action recognition performance is improved, but computational resources and training costs increase significantly
Solution Approach 1:
The patent segments the action recognition task by creating a taxonomy of human activities that divides the broad task into specific sub-tasks. The training data is then organized according to this taxonomy, allowing the model to learn hierarchical representations of actions. This segmentation enables the model to achieve high performance on specific action recognition tasks without requiring massive computational resources for all possible actions simultaneously.
Solution Approach 2:
The patent applies local quality by generating domain-specific training data focused on human activities and actions, rather than using generic large-scale data. The method creates tailored image-text pairs and captions specifically for action recognition, concentrating computational resources on high-quality, relevant data rather than diluting them across vast amounts of unrelated data.
2Productivity
If domain-specific training data is generated using LLMs, then training efficiency and accuracy are improved, but data generation time and computational cost increase
Solution Approach 1:
The patent performs preliminary actions by pre-building a taxonomy of human activities and pre-generating domain-specific queries before the main training process. The LLM is used to create the hierarchical structure and generate queries in advance, which then guide the retrieval and generation processes during training, reducing the time needed for these operations during actual model training.
Solution Approach 2:
The patent introduces an intermediary approach by using the LLM-generated taxonomy and queries as mediators between the raw data and the training process. Instead of directly processing all raw data, the system uses the generated queries and structured representations as intermediaries to filter, select, and guide the training data, making the data generation and processing more efficient.
Data Source
Figure 1(a)~1(b)
Figure 2~3
Figure 4
AI summary
Broadly speaking, embodiments of the present techniques provide a method for generating a training dataset suitable for training a vision-language, VL, machine learning, ML, model for performing action recognition.