Visual Subtitle Generation via Viseme Mapping and Audio Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies lack effective methods for accurately inserting visual subtitles in videos, particularly for aiding the hearing impaired, and fail to synchronize visual and auditory cues for enhanced speech perception.
Innovation Solution
A system and method that segments audio signals to identify speakers, recognizes phonemes, maps them to visemes, and integrates animated lip and eyebrow movements into videos, creating synchronized visual subtitles to enhance speech interpretation and emotion detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If visual subtitles are inserted to aid hearing impaired users, then accessibility and speech perception are improved, but video complexity and processing requirements increase
Solution Approach 1:
The system segments the video into multiple tracks including original video, visual subtitles, and audio. This segmentation allows independent processing of each component, enabling the insertion of visual subtitles without reprocessing the entire video, thus improving speech perception while managing processing complexity through modular handling of video components.
Solution Approach 2:
The patent creates visual subtitles as a separate copy layer that overlays on the original video. Instead of modifying the original video directly, the system generates a parallel visual representation of speech that can be superimposed, maintaining the original video integrity while adding accessibility features through a copied visual track.
2Measurement precision
If visual and auditory cues are synchronized for enhanced speech perception, then speech interpretation accuracy is improved, but timing precision and synchronization complexity increase
Solution Approach 1:
The system performs preliminary synchronization by aligning visual subtitles with audio tracks during the video processing stage. Timestamps and synchronization metadata are embedded in advance, allowing the player to automatically sync visual and auditory cues without real-time processing delays, thus improving speech interpretation accuracy while managing synchronization precision through pre-computed alignment data.
3Measurement precision
If multiple visual subtitle tracks are created for different speakers, then speech attribution accuracy is improved, but video file size and processing load increase
Solution Approach 1:
The patent merges multiple speaker-specific visual subtitle tracks into a single composite visual subtitle stream. Instead of maintaining separate tracks for each speaker, the system combines them with speaker attribution metadata, reducing the overall data volume while preserving speaker identification accuracy through integrated multi-speaker visual representation.
Solution Approach 2:
The visual subtitle system is designed to handle multiple speakers universally through a single track structure that can accommodate different speakers. The visual subtitles use universal formatting and attribution mechanisms that work across multiple speakers, eliminating the need for separate dedicated tracks for each speaker and reducing overall data volume while maintaining speaker identification capability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system and method to insert visual subtitles in videos is described. The method comprises segmenting an input video signal to extract the speech segments and music segments. Next, a speaker representation is associated for each speech segment corresponding to a speaker visible in the frame. Further, speech segments are analysed to compute the phones and the duration of each phone. The phones are mapped to a corresponding viseme and a viseme based language model is created with a corresponding score. Most relevant viseme is selected for the speech segments by computing a total viseme score. Further, a speaker representation sequence is created such that phones and emotions in the speech segments are represented as reconstructed lip movements and eyebrow movements. The speaker representation sequence is then integrated with the music segments and super imposed on the input video signal to create subtitles.