Multimodal Video Query Response Using Attention-Weighted Shot Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video story query response technologies face challenges in handling the complexity of video data, failing to effectively utilize audio information and emotions, leading to limited dataset collection and inaccurate query responses.
Innovation Solution
A query response device and method that divides video data into shots, extracts feature vectors from image frames, audio, and captions, and uses a multilayer neural network with attention weights to select appropriate responses based on similarities between video and option feature vectors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional video query response models use only caption and image frame information, then the model complexity is reduced, but the response accuracy deteriorates because speaker and emotion information from audio is not utilized
Solution Approach 1:
The patent segments video data into multiple modalities (image frames, audio data, caption data) and processes each through separate neural networks before integration. This segmentation allows comprehensive feature extraction from all sources including audio speaker and emotion information, improving response accuracy while managing complexity through modular architecture
Solution Approach 2:
The patent merges features from multiple neural networks processing different modalities (image, audio, caption) into a unified response generation process. By combining these feature sets, the system achieves more accurate responses that incorporate speaker identification and emotion detection from audio alongside visual and textual information
2Productivity
If video data is processed without hierarchical modeling, then the processing speed is improved, but the ability to handle complex video stories deteriorates
Solution Approach 1:
The patent segments video content into shot-level units and processes them hierarchically through multiple neural network layers. This segmentation enables efficient processing of individual shots while building up to comprehensive video story understanding, maintaining processing speed through localized operations rather than global analysis
Solution Approach 2:
The patent introduces hierarchical dimensionality by processing video data at multiple levels (shot-level features, scene-level aggregation, story-level synthesis). This dimensional approach enables the system to handle complex video stories effectively while maintaining processing efficiency through progressive feature building
3Adaptability or versatility
If audio information is not integrated with image and caption data, then the data processing complexity is reduced, but the ability to identify speaker and emotion deteriorates
Solution Approach 1:
The patent segments audio data processing into a separate neural network branch that independently extracts speaker and emotion features before integrating with image and caption features. This segmentation enables specialized audio processing for speaker and emotion identification without overwhelming the overall system complexity
Solution Approach 2:
The patent uses an intermediary fusion mechanism that combines audio-derived speaker and emotion features with visual and textual features. This intermediary layer integrates multi-modal information effectively, enabling accurate speaker and emotion identification while managing processing complexity through structured feature fusion
Data Source
AI summary
A query response method includes: dividing image frames, audio data and caption data included in video data of a data set on a per-shot basis based on the same single caption; extracting a shot feature vector by calculating the feature vectors of image frames, audio data and caption data included in each shot; extracting feature vectors of query data and a plurality of pieces of option data corresponding to the query data from each query-response pair included in the data set; calculating a video feature vector by inputting the shot feature vectors into a multilayer neural network, assigning an attention weight, calculated based on the feature vector of the query data, to output vectors of respective layers, and then summing the weighted output vectors; and selecting a final response from among the plurality of pieces of option data based on similarities between the video feature vector and option feature vectors.