Multimedia Similarity Detection with Mel Spectrogram Audio Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multimedia information processing systems struggle to accurately identify similarities between videos that have undergone complex editing, such as excessive cropping or audio manipulation, leading to low similarity identification accuracy and potential copyright infringement.
Innovation Solution
A multimedia information processing method involving audio feature extraction using Mel spectrograms and a VGGish network to determine audio feature vectors, followed by similarity analysis through a triplet-loss layer network, to enhance identification accuracy and detect potential copyright infringements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If video image fingerprints are used for similarity identification, then the process is simple, but the accuracy is low when videos have been excessively cropped or edited
Solution Approach 1:
The patent segments the video into multiple audio frames and extracts audio features from each frame independently. Instead of treating the entire video as a single unit, the system divides it into manageable segments (frames) and processes each segment to extract acoustic features, which are then aggregated for similarity comparison. This segmentation approach maintains robustness against video editing while preserving accuracy.
Solution Approach 2:
The patent replaces the mechanical/video-based fingerprint system with an acoustic/audio-based system. By substituting visual analysis with auditory analysis, the system becomes insensitive to visual transformations like cropping, filtering, or visual effects, while maintaining the ability to detect similarity through audio content that remains unchanged by such edits.
2Measurement precision
If audio feature extraction is added to improve accuracy, then the identification accuracy improves, but the processing complexity increases
Solution Approach 1:
The patent implements a multi-functional system that can handle both video-based and audio-based similarity identification. The same framework processes different types of features (visual and acoustic) using unified similarity comparison logic, making the system versatile and reducing overall complexity despite the added audio processing capability.
Solution Approach 2:
The patent introduces an intermediary audio feature extraction layer that bridges the original video content and the similarity comparison process. This intermediary layer converts complex video data into simplified acoustic feature representations, which then feed into the similarity identification system, reducing the complexity of direct video comparison while maintaining accuracy.
3Measurement precision
If deep metric learning with triplet loss is used, then the feature discrimination capability is enhanced, but the training data requirements and computational resources increase
Solution Approach 1:
The patent applies parameter changes by modifying the loss function parameters in the neural network training process. By adjusting the triplet loss parameters (margins, weights, and thresholds) and using cross-entropy loss with carefully selected class labels, the system achieves effective feature discrimination without requiring excessively large training datasets. The parameter optimization allows the model to learn robust acoustic features from limited training data.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
This application provides a multimedia information processing method and apparatus, an electronic device, and a storage medium. The method includes: parsing multimedia information to separate an audio from the multimedia information; converting the audio to obtain a mel spectrogram corresponding to the audio; determining, according to the mel spectrogram corresponding to the audio, an audio feature vector corresponding to the audio; and determining, based on an audio feature vector corresponding to a source audio in source multimedia information and an audio feature vector corresponding to a target audio in target multimedia information, a similarity between the target multimedia information and the source multimedia information.