Multimodal Term Weight Generation for Video Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing term weight generation methods in video search systems are inaccurate in distinguishing between key terms and confusing terms in video titles, as they primarily rely on text-based features without incorporating video-related information.
Innovation Solution
A method that performs multimodal feature fusion of video, video-associated text, and terms to generate a halfway vector, which is then used to calculate the weight of each term, incorporating both text and video features for improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If text-based feature encoding is used for term weight generation, then the process is simple and fast, but the accuracy in distinguishing key terms from confusing terms is insufficient
Solution Approach 1:
The patent merges text features and video features into a unified feature space through multimodal feature fusion. The text encoder and video encoder both map their respective inputs to embedding vectors that are then combined to generate term weights, allowing the system to leverage both textual and visual information for more accurate term weight generation.
Solution Approach 2:
The patent employs universal encoders that can process different types of inputs (text and video) and transform them into a common representation space. The text encoder handles video titles and descriptions, while the video encoder processes video frames, both producing embedding vectors that feed into the same term weight generation mechanism.
2Reliability
If video features are incorporated into term weight generation, then the distinction between key terms and confusing terms improves, but the computational complexity increases
Solution Approach 1:
The patent performs preliminary encoding of video frames into feature vectors before the term weight generation process. The video encoder pre-processes the video content and stores it as embedding vectors, which are then readily available for fusion with text features during search operations, avoiding redundant computation.
Solution Approach 2:
The patent segments the video content into individual frames that are independently encoded into feature vectors. This segmentation allows the system to process video content in manageable units and selectively use relevant frame features for term weight generation, reducing overall computational burden.
Data Source
AI summary
A term weight determination method includes: obtaining a video and video-associated text, the video-associated text including at least one term; generating a halfway vector of the term by performing multimodal feature fusion on the features of the video, the video-associated text and the at least one term; and generating the weight of the at least one term based on the halfway vector of the at least one term.