Video Mixing Using Speech Detection to Reduce Processing Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video conferencing systems face inefficiencies in processing resources due to the need to decode all incoming media streams to determine which to mix and relay, and are prone to false detections of speaking participants, especially in the presence of background noise.
Innovation Solution
Implementing video-based speech activity detection at remote clients to select and transmit media frames based on video speech activity signals, allowing for efficient selection and mixing of media streams without decoding at the server, thereby reducing processing load and improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If voice activity detection is used to determine which media streams to mix, then the server can identify speaking participants, but the server must decode all incoming media streams which consumes significant processing resources
Solution Approach 1:
The patent introduces video-based speech activity detection as an intermediary mechanism between the media stream and the mixing decision. Instead of directly decoding audio streams for VAD, the system uses video frames as an intermediary to detect speech activity, which consumes fewer processing resources while maintaining detection accuracy.
Solution Approach 2:
The patent substitutes the traditional audio-based VAD mechanism with a video-based detection mechanism. By replacing audio signal processing with video frame analysis, the system achieves speech detection without the need to decode all incoming media streams, thereby reducing processing resource expenditure.
2Productivity
If voice activity detection is performed on all incoming media streams, then the server can determine which streams to relay, but false detections occur in the presence of background noise
Solution Approach 1:
The patent replaces audio-based VAD with video-based speech activity detection. Video-based detection analyzes visual cues such as lip movement and facial expressions, which are less susceptible to background noise interference compared to audio-based methods, thereby reducing false detections while maintaining efficient media stream selection.
3Measurement precision
If the server decodes all incoming media streams to determine mixing requirements, then accurate speech detection is achieved, but processing time and resources are significantly increased
Solution Approach 1:
The patent extracts the speech activity detection function from the media stream decoding process. By using video frames that are already being processed for other purposes, the system can determine speech activity without extracting and decoding the entire audio stream, thereby reducing processing time while maintaining detection precision.
Solution Approach 2:
The patent performs speech activity detection using video frames that are processed in advance for other video conferencing tasks. By conducting detection before or during the normal video processing pipeline, the system avoids the time-consuming step of decoding all audio streams while still achieving accurate speech detection.
Data Source
AI summary
A method for video conferencing includes receiving, at one or more computers from at least some remote clients from a plurality of remote clients, information representing a plurality of media frames. The method also includes receiving, at the one or more computers from at least some of the plurality of remote clients, a plurality of video-based speech activity signals each associated with a respective media frame from the plurality of media frames. The method further includes selecting, at the one or more computers, at least some media frames from the plurality of media frames based on the video-based speech activity signals, decoding the selected media frames, generating a mixed media stream by combining the decoded media frames, transmitting, from the one or more computers to at least some remote clients from the plurality of remote clients, the mixed media stream.