Multi-modal Lie Detection via High-order Feature Interaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional lie detection methods using contact-type physiological signals are limited by coordination and fail to accurately detect lies, while non-contact methods based on audio and video features stitched by deep neural networks neglect high-order interactions, leading to unsatisfactory lie detection results.
Innovation Solution
A multi-modal lie detection method utilizing recurrent neural networks and a self-attention mechanism for feature extraction and context modeling, which explicitly depicts first-order, second-order, and third-order interactions between audio, video, and text features to improve lie classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If features of different modalities are simply stitched together, then the processing complexity is reduced, but the lie detection accuracy deteriorates due to neglecting high-order interactive relationships
Solution Approach 1:
The patent applies composite material principle by integrating multiple feature representation methods (original features, first-order interactions, second-order interactions, third-order interactions) to form a composite feature vector. This composite structure captures high-order interactive relationships between audio, video, and text modalities while maintaining manageable processing complexity through systematic organization of interaction terms.
2Duration of action of moving object
If recurrent neural network is used for context modeling, then sequential dependencies are captured, but the long-term modeling effect deteriorates leading to unsatisfying lie detection results
Solution Approach 1:
The patent introduces self-attention mechanism as an intermediary component that works in conjunction with the recurrent neural network. The self-attention mechanism enables the model to directly access and weigh important information from any position in the sequence, acting as a mediator that complements the RNN's sequential processing and significantly improves long-term dependency modeling for lie detection.
3Ease of operation
If non-contact lie detection methods based on audio and video are used, then the coordination requirement is eliminated, but the lie detection accuracy deteriorates due to simple feature stitching
Solution Approach 1:
The patent transitions from simple feature stitching (one-dimensional concatenation) to multi-dimensional feature interaction by explicitly modeling first-order, second-order, and third-order interactions between audio, video, and text modalities. This dimensional expansion captures complex cross-modal relationships that simple stitching cannot represent, thereby improving lie detection accuracy while maintaining ease of operation.
Data Source
AI summary
A multi-modal lie detection method and apparatus, and a device to improve an accuracy of an automatic lie detection are provided. The multi-modal lie detection method includes inputting original data of three modalities, namely a to-be-detected audio, a to-be-detected video and a to-be-detected text; performing a feature extraction on input contents to obtain deep features of the three modalities; explicitly depicting first-order, second-order and third-order interactive relationships of the deep features of the three modalities to obtain an integrated multi-modal feature of each word; performing a context modeling on the integrated multi-modal feature of the each word to obtain a final feature of the each word; and pooling the final feature of the each word to obtain global features, and then obtaining a lie classification result by a fully-connected layer.