Interpolated Translatable Audio for Moving Avatars Under Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing online virtual experience platforms face challenges in providing immersive audio due to latency issues, where audio mixing at cloud servers results in audio output that is noticeably wrong because the client device's avatar has moved, causing a mismatch between sound emitters and the listener's position.
Innovation Solution
The method involves predicting the avatar's subsequent positions using techniques like dead-reckoning, input-based prediction, game-logic, Kalman filters, or machine-learning models, sampling audio mixes at interpolation regions, and transmitting packets to the client device for interpolation to ensure accurate audio output.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If audio mixing is performed at cloud servers for virtual experience platforms, then the number of simultaneous sound sources can be increased, but latency is introduced causing audio output to be inaccurate when the avatar moves to new positions
Solution Approach 1:
The system pre-calculates and transmits audio mixes for multiple predicted avatar positions before the avatar actually reaches those positions. By predicting future positions and preparing audio content in advance, the system eliminates latency-induced audio inaccuracies while maintaining the ability to handle many simultaneous sound sources through cloud-based mixing
2Reliability
If audio mixing is performed at cloud servers, then audio quality for multiple users can be maintained, but the avatar's movement during transmission causes mismatch between sound emitters and listener position
Solution Approach 1:
The system dynamically adjusts audio mixes based on predicted avatar positions rather than using static pre-mixed audio. By continuously updating audio content to match the avatar's movement trajectory, the system maintains both audio quality consistency and position accuracy simultaneously
Solution Approach 2:
The system uses avatar position feedback to determine which pre-calculated audio mixes to transmit and when. By monitoring the avatar's actual position and comparing it with predicted positions, the system selects the most appropriate audio mix, ensuring both quality consistency and position accuracy
Data Source
AI summary
According to one aspect, a method of audio-signal processing of a network device is described. The method may include obtaining an audio portion associated with a sound field of an avatar at an initial position in a virtual experience. The method may include identifying at least one interpolation region associated with the avatar at the initial position or associated with the avatar at one or more subsequent positions. The method may include sampling an audio mix for a plurality of points of the at least one interpolation region based on the audio portion associated with the sound field of the avatar at the initial position or associated with the sound field of the avatar at the one or more subsequent positions. The method may include transmitting an audio packet associated with the audio mix to a client device.