AR Experience Audio Capture With Separate Virtual Sound Mixing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing audio capture technologies in augmented reality (AR) devices fail to accurately reproduce the sound as perceived by the user, with the audio playback often distorted due to low-quality mixing of AR-generated sounds and real-world sounds, and synchronization challenges between speaker output and microphone recording.
Innovation Solution
A processor-based system that digitally captures AR audio before output by the speaker, using techniques like echo cancellation and volume ducking to enhance audio quality, and allows for separate storage and later mixing of acoustic signals and AR audio to improve sound fidelity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If AR audio is captured through the speaker-to-microphone path, then the system can record virtual sound objects, but the audio quality deteriorates due to low-fidelity reproduction and mixing issues
Solution Approach 1:
The system performs preliminary action by capturing AR audio content digitally before it is output through the speaker. The audio processor receives the AR audio content from the AR application and stores it separately, eliminating the need to capture it through the speaker-to-microphone path. This preliminary digital capture preserves audio quality while avoiding the fidelity losses and mixing issues that would otherwise occur.
2Reliability
If real-time mixing is performed between AR audio and acoustic signals, then synchronization can be maintained, but audio quality deteriorates due to the speaker-to-microphone capture path
Solution Approach 1:
The system performs preliminary action by capturing and storing the AR audio content digitally before it is output through the speaker. This pre-captured digital audio is then mixed with the acoustic signals captured by the microphone array during post-processing, rather than attempting real-time mixing through the speaker-to-microphone path. This approach maintains both synchronization reliability and audio fidelity.
Solution Approach 2:
The system introduces an intermediary approach by using a separate digital audio stream for AR content that acts as a mediator between the AR application and the final mixed output. Instead of forcing AR audio through the acoustic capture path, the system uses this intermediate digital representation to combine with microphone recordings during post-processing, preserving audio quality while enabling synchronization.
3Measurement precision
If AR audio volume is increased to match real-world sounds, then virtual sound objects become audible, but distortion occurs due to volume ducking and mixing issues
Solution Approach 1:
The system performs preliminary action by capturing the AR audio content at its original, undistorted volume level directly from the AR application before output. This pre-captured digital audio maintains the correct volume relationship with real-world sounds, eliminating the need for volume ducking and subsequent distortion that would occur if the audio were captured through the speaker-to-microphone path and then amplified.
Data Source
AI summary
A method to enhance virtual audio capture in Augmented Reality (AR) experience recordings starts with a processor receiving a video from a camera that includes images of a real-world scene and an AR content item. Processor receives acoustic signals from microphones generate acoustic signals using real-world audio and speaker output that including AR audio of the AR content item. Processor receives an audio file associated with the AR audio of the AR content item and generates an enhanced audio using the acoustic signals and the audio file. Processor generates an enhanced video using the video and the enhanced audio. Other examples are described herein.