Multimodal Lie Detection Using Graph Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automatic lie detection methods fail to accurately detect lies by neglecting the influence of previous lie states and interactive information between speakers during interrogations, leading to inaccurate results.
Innovation Solution
An automatic lie detection method that segments video, audio, and text into short-term features, aggregates them chronologically, integrates these features using a multi-modal attention strategy, and inputs them into a graph neural network with speaker information to predict lie levels, considering sequential and interactive information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If only current word facial expressions and speeches are analyzed, then the analysis process is simple, but the detection accuracy is low due to neglecting previous lie states and interactive information
Solution Approach 1:
The patent segments the analysis process into multiple time periods, extracting short-term features for each period and then aggregating them chronologically to form long-term features. This segmentation allows the system to consider temporal sequences and interactive information while maintaining manageable processing stages.
Solution Approach 2:
The patent introduces a temporal dimension by aggregating short-term features chronologically into long-term features, and another dimension by integrating multi-modal features (video, audio, text) through attention mechanisms. This dimensional expansion enables comprehensive analysis of previous lie states and interactive information.
2Measurement precision
If multi-modal features with interactive and sequential information are integrated, then the detection accuracy is improved, but the computational complexity increases
Solution Approach 1:
The patent performs preliminary feature extraction for each modality (video, audio, text) separately at the short-term level, organizing features by time periods before aggregation. This preliminary organization reduces computational complexity by preparing data in advance for the more intensive aggregation and integration operations.
Solution Approach 2:
The patent uses attention mechanisms as intermediaries to selectively weight and integrate multi-modal features. The attention mechanism efficiently combines video, audio, and text features by focusing on the most relevant information, reducing the computational burden of processing all features equally.
Data Source
AI summary
An automatic lie detection method and apparatus for interactive scenarios, a device and a medium to improve the accuracy of automatic lie detection are provided. The method includes: segmenting three modalities, namely a video, an audio and a text, of a to-be-detected sample; extracting short-term features of the three modalities; integrating the short-term features of the three modalities in the to-be-detected sample to obtain long-term features of the three modalities corresponding to each dialogue; integrating the long-term features of the three modalities by a self-attention mechanism to obtain a multi-modal feature of the each dialogue; integrating the multi-modal feature of the each dialogue with interactive information by a graph neutral network to obtain a multi-modal feature integrated with the interactive information; and predicting a lie level of the each dialogue according to the multi-modal feature integrated with the interactive information.