Mixed Voice Chat Muting Using Spatial Synthetic Audio
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In virtual environments, muting an abusive participant's audio stream is challenging because mixing all streams together makes it difficult to individually mute specific participants without precise timing and quality matching, undoing the efficiency of creating a mixed stream.
Innovation Solution
A client device generates additional audio that matches the location and orientation of the muted participant, mixing it with the encoded audio to drown out their voice, using synthetic speech or other sounds to effectively mute the abusive stream.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If all audio streams are mixed together and sent as a single mixed stream, then network bandwidth efficiency is improved (scaling from N^2 to N streams), but the ability to individually mute specific participants is lost
Solution Approach 1:
The patent segments the audio processing into two parts: (1) server-side mixing of all audio streams into a single mixed stream for efficient transmission, and (2) client-side separation where the mixed stream is split into individual participant streams using spatial audio processing and head-related transfer functions (HRTFs). This segmentation allows the server to benefit from reduced bandwidth while enabling clients to individually mute specific participants by filtering out their isolated audio streams.
Solution Approach 2:
The patent introduces spatial audio processing and HRTFs as an intermediary layer between the mixed stream and individual participant streams. The mixed audio is processed through spatial rendering that assigns each participant's voice to a specific spatial location, creating an intermediate representation that preserves individual stream information while maintaining the efficiency of mixed stream transmission. This intermediary enables selective muting without requiring the server to transmit separate streams.
2Ease of operation
If individual audio streams are transmitted to all clients, then individual muting capability is improved, but network bandwidth consumption increases significantly
Solution Approach 1:
The patent segments the audio processing into two parts: (1) server-side mixing of all audio streams into a single mixed stream for efficient transmission, and (2) client-side separation where the mixed stream is split into individual participant streams using spatial audio processing and head-related transfer functions (HRTFs). This segmentation allows the server to benefit from reduced bandwidth while enabling clients to individually mute specific participants by filtering out their isolated audio streams.
Solution Approach 2:
The patent implements self-service by enabling each client to perform local audio processing to separate and filter individual participant streams from the mixed stream. Instead of requiring the server to provide individual streams to all clients (which would consume excessive bandwidth), each client independently processes the mixed stream locally using spatial audio algorithms, generating the necessary individual stream separations autonomously at the client device.
3Productivity
If audio streams are mixed together, then processing efficiency is improved, but the precision of identifying and muting specific participants deteriorates
Solution Approach 1:
The patent introduces spatial audio processing and HRTFs as an intermediary layer between the mixed stream and individual participant streams. The mixed audio is processed through spatial rendering that assigns each participant's voice to a specific spatial location, creating an intermediate representation that preserves individual stream information while maintaining the efficiency of mixed stream transmission. This intermediary enables selective muting without requiring the server to transmit separate streams.
Solution Approach 2:
The patent applies local quality by associating each participant's audio with specific spatial characteristics (direction, distance, position in 3D space) rather than treating all audio uniformly. This spatial tagging allows the system to maintain processing efficiency through mixing while enabling precise participant identification and selection based on spatial attributes, so that muting operations can target specific spatial locations corresponding to specific participants.
Data Source
AI summary
A metaverse application receives encoded audio that includes a first audio stream associated with a first avatar in a virtual environment and a first voice-activity detection (VAD) signal for the first audio stream, and a second audio stream associated with a second avatar in the 3D virtual environment and a second VAD signal for the second audio stream. The metaverse application determines that the first avatar is blocked by a user associated with the user avatar. The metaverse application determines that the first VAD signal indicates that the first audio stream includes speech. The metaverse application generates additional audio. The metaverse application mixes the additional audio with the encoded audio. The metaverse application provides the mixed audio to a speaker for output.