Deepfake Video Detection Using Face and Voice Direction Mismatch

The patent introduces face direction analysis as an intermediary element between the video content and voice analysis. By extracting face direction information from video frames and comparing it with voice direction information derived from audio signals, the system creates an additional verification layer that is independent of lip-syncing manipulation, thereby improving detection reliability while maintaining implementation feasibility

The patent replaces the mechanical lip-syncing verification method with an acoustic-based voice direction analysis method. By using Direct to Reverberant speech energy ratio (DRR) to infer voice direction from audio signals and comparing it with face direction from video, the system substitutes the vulnerable lip-syncing mechanism with a more reliable acoustic field-based approach

The patent segments the deepfake detection task into distinct modules: face direction extraction from video frames using pre-trained deep learning models, voice direction extraction from audio signals using DRR analysis, and comparison of the two directions. This segmentation allows each module to be optimized independently and enables parallel processing, reducing overall computational complexity while maintaining high detection accuracy

The patent applies pre-trained deep learning models for object detection that have already been trained on large datasets. By using pre-trained models rather than training new models from scratch, the system achieves high detection accuracy while significantly reducing the computational resources and time required for model training, thus lowering the overall computational complexity