Distributed Active Speaker Detection for Frontal Teleconference Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conference room video cameras have limitations in capturing high-quality, frontal views of users due to their placement, which restricts the user experience in teleconferencing.
Innovation Solution
Employ active speaker detection on user devices to dynamically switch to a high-resolution view from the user's device camera when they speak, using personalized enhancement models to isolate their voice and enhance the audio experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a conference room video camera is used to capture users, then the system can provide video signals to remote devices, but the camera placement limits the ability to capture high-quality frontal views of users
Solution Approach 1:
The system divides the video capture function between the conference room camera (for overall scene capture) and individual user devices (for personal frontal views). Each user's device captures its own frontal video when the user speaks, segmenting the single camera function into multiple distributed camera functions to achieve both overall context and high-quality individual views.
Solution Approach 2:
The active speaker detection system acts as an intermediary that coordinates between the conference room camera and user devices. It detects when a user is speaking and triggers the corresponding user device to capture and transmit its video feed, mediating the interaction between multiple video sources to provide high-quality frontal views on demand.
2Quantity of substance
If all users' video signals are transmitted simultaneously, then all participants are visible, but bandwidth consumption increases
Solution Approach 1:
Instead of continuously transmitting all user video feeds, the system uses periodic action by triggering video capture and transmission only when a user is detected to be speaking. This intermittent transmission based on speaking detection significantly reduces bandwidth consumption while maintaining relevant video content availability.
Solution Approach 2:
The system transmits video signals selectively rather than completely - only for users who are actively speaking rather than all users simultaneously. This partial action approach reduces the total volume of video data transmitted while maintaining the quality of communication during active speaking periods.
3Device complexity
If conference room camera is used for all users, then infrastructure is simple, but individual user engagement and visibility are reduced
Solution Approach 1:
User devices serve multiple functions: they act as both the user's personal camera for capturing frontal views and as their audio device for speaking detection. This multi-functionality eliminates the need for separate dedicated cameras at each user position while achieving high-quality individual views and maintaining system simplicity.
Data Source
AI summary
This document relates to active speaker detection using distributed devices. For example, the disclosed implementations can employ personal devices of one or more users to detect when those users are speaking during a call with other users. Then, a camera on the personal device can be employed to obtain a front-facing view of the user, which can be provided to other call participants. In some cases, a microphone and/or camera on the user's device are employed to detect when the user is actively speaking.