Missing Subtitle Detection via Neural Network Audio Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing media presentation technologies often result in missing subtitles during content playback, particularly when foreign dialogs are present, leading to an unpleasant viewer experience due to human transcription errors.
Innovation Solution
A computer-implemented method using a voice activity detection (VAD) network and an audio classification (AC) network to identify speech segments and classify sounds, allowing for the detection of missing subtitles by processing audio sequences into segments and combining them to filter out misidentifications, thereby generating notifications for missing subtitles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If human transcription is used to create subtitles, then the subtitle content can be generated, but errors occur causing missing subtitles particularly in foreign dialogs
Solution Approach 1:
An audio classification network is introduced as an intermediary between the audio content and subtitle generation process. This intermediary automatically classifies audio segments as speech or non-speech, providing an additional layer of verification that complements human transcription and identifies missing foreign dialogs that humans may overlook.
Solution Approach 2:
The patent replaces part of the manual human transcription process with an automated audio classification system using machine learning. This substitution handles the specific task of identifying speech segments, reducing reliance on human translators and eliminating their tendency to miss foreign language content.
2Difficulty of detecting and measuring
If audio segments are processed individually by VAD network, then speech detection can be performed, but misidentifications occur such as coughing being detected as speech
Solution Approach 1:
The patent combines multiple consecutive speech segment predictions into larger audio segments before final classification. This merging process allows the system to aggregate weak signals and filter out isolated false positives (like coughing), while preserving genuine speech patterns that span multiple segments through temporal consistency.
3Reliability
If multiple consecutive speech segments are combined, then misidentifications can be filtered out, but the processing complexity increases
Solution Approach 1:
The audio stream is segmented into fixed-duration chunks that are processed in sequence. This segmentation approach breaks down the complex task of continuous audio analysis into manageable discrete units, making it computationally feasible to combine multiple segments while maintaining systematic processing and reducing overall complexity.
Data Source
AI summary
Some implementations include methods for detecting missing subtitles associated with a media presentation and may include receiving an audio component and a subtitle component associated with a media presentation, the audio component including an audio sequence, the audio sequence divided into a plurality of audio segments; evaluating the plurality of audio segments using a combination of a recurrent neural network and a convolutional neural network to identify refined speech segments associated with the audio sequence, the recurrent neural network trained based on a plurality of languages, the convolutional neural network trained based on a plurality of categories of sound; determining timestamps associated with the identified refined speech segments; and determining missing subtitles based on the timestamps associated with the identified refined speech segments and timestamps associated with subtitles included in the subtitle component.