Question Answering Apparatus Character-Centered Attention Mechanism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video story question-answering technologies face difficulties due to the complexity of video data and the inability to effectively utilize information about utterers and emotions, which are not inferred from subtitles or images, leading to limitations in data sets and question-answer models.
Innovation Solution
A method involving character-centered modeling that generates input time-series sequences from video content and question-answer pairs, calculates weights, and uses attention mechanisms to select final answers by concatenating relevant sequences and Boolean flags, enabling comprehensive understanding of video stories through a dual matching multi-stream model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional video question-answer models use only subtitle and image frame information, then the model complexity is reduced, but the ability to utilize information about utterers and emotions is insufficient
Solution Approach 1:
The patent segments video information into multiple independent streams: visual stream (image frames), audio stream (subtitles and voice), and character stream (utterer identification). Each stream is processed separately through dedicated neural network layers, allowing comprehensive information extraction without excessive model complexity. The character-centered annotations are also segmented into multiple difficulty levels for hierarchical processing.
Solution Approach 2:
The patent adds a new dimension of character-centered modeling by introducing character embedding vectors and utterer identification mechanisms. This transforms the traditional 2D (image-frame based) processing into a 3D multi-stream processing framework that incorporates temporal, spatial, and character dimensions simultaneously, enabling effective utilization of utterer and emotion information.
2Loss of information
If video story question answering uses comprehensive video data including audio and multiple frames, then the information utilization is improved, but the difficulty of data collection and processing increases
Solution Approach 1:
The patent divides comprehensive video data into separate processing streams: visual features from image frames, audio features from subtitles and voice signals, and character features from utterer identification. Each stream has dedicated extraction and processing modules, making the handling of complex multi-modal data more manageable and systematic.
Solution Approach 2:
The patent performs preliminary processing of video data by pre-extracting visual features, audio features, and character features before the main question-answering process. Character-centered annotations are pre-computed and stored, reducing the computational burden during actual inference and simplifying the data processing pipeline.
3Measurement precision
If character-centered modeling with hierarchical annotations is implemented, then the question answering accuracy is improved, but the data annotation complexity increases
Solution Approach 1:
The patent segments character-centered annotations into hierarchical levels: basic character identification, character relationship identification, and character emotion/role identification. This hierarchical segmentation allows annotators to work systematically from simple to complex tasks, improving annotation accuracy while making the process more manageable.
Solution Approach 2:
The patent introduces character embedding vectors as intermediary representations that bridge raw video data and question-answering tasks. These embeddings serve as a standardized intermediate format that simplifies the annotation process by providing a consistent structure for character information across different video contexts.
Data Source
AI summary
A question answering method that is performed by a question answering apparatus includes: receiving a data set including video content and question-answer pairs; generating input time-series sequences from the video content of the input data set and also generating a question-answer time-series sequence from the question-answer pair of the input data set; calculating weights by associating the input time-series sequence with the question-answer time-series sequence and also calculating first result values by performing operations on the calculated weights and the input time-series sequences; calculating second result values by paying attention to portions of the input time-series sequences that are directly related to characters appearing in questions and answers; and calculating third result values by concatenating the time-series sequences, the first result values, the second result values, and Boolean flags and selecting a final answer based on the third result values.