Animated Sign Language Avatars for Multi-Speaker Video Sessions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Individuals with reduced or no ability to perceive audio output face challenges in comprehending video content, particularly in discerning between speakers and maintaining attention to visual output simultaneously.
Innovation Solution
Generating animated avatars that provide sign language content by determining spoken words, locating speakers, and modifying video data to include avatars that correspond to speakers, allowing simultaneous viewing of both speakers and avatars without diverting attention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If sign language content is provided to help individuals understand video content, then comprehension ability is improved, but device complexity increases due to the need for avatar generation and video modification systems
Solution Approach 1:
An animated avatar is introduced as an intermediary element that translates spoken words into sign language and visually presents them alongside the video content. This mediator bridges the gap between audio-based video content and visual-based comprehension for hearing-impaired individuals, converting information from one modality to another without requiring complex direct modification of the original video
Solution Approach 2:
Instead of fundamentally transforming the video structure, the system creates a visual copy or representation of the spoken content through the animated avatar. The avatar replicates the information content of the audio track in sign language form, providing a parallel visual channel that preserves the original video while adding complementary information
2Loss of information
If multiple speakers are present in the video, then information diversity is improved, but difficulty in discerning speakers increases for hearing-impaired individuals
Solution Approach 1:
Each animated avatar is given distinct visual characteristics or is positioned in specific locations corresponding to their respective speakers. This local differentiation allows hearing-impaired individuals to easily distinguish which avatar represents which speaker, maintaining speaker identity information through spatial and visual quality variations rather than audio cues
3Ease of operation
If alternative visual features are provided for hearing-impaired individuals, then accessibility is improved, but ability to maintain attention to visual output simultaneously deteriorates
Solution Approach 1:
The animated avatar is integrated directly into the video frame and merged with the existing visual content rather than being presented as a separate overlay or caption layer. This unified visual presentation combines the original video and sign language representation into a single coherent visual field, reducing the cognitive load of switching between multiple visual channels while maintaining accessibility
Data Source
AI summary
A method includes receiving, via one or more processors, video data and audio data associated with respective participants in a video session, determining, via the one or more processors, words spoken by a speaker participant in the video session based on the audio data, determining, via the one or more processors, a location of the speaker participant in a framing of the video data based on the video data and the audio data, generating, via the one or more processors, an animated avatar to provide sign language representing the words spoken by the speaker participant, modifying, via the one or more processors, the video data of the video session to include the animated avatar based on the location of the speaker participant, and outputting, via the one or more processors, the video data that includes the animated avatar.