Surface-Appropriate Collision Audio for Mixed Reality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing XR systems fail to accurately present virtual audio that matches a user's surroundings and spatial movements of virtual objects, leading to auditory inconsistencies that disrupt immersion and can cause motion sickness.
Innovation Solution
Systems and methods for generating audio signals based on a trigger event, such as a footstep, using a wearable head device with a microphone and sensor to identify a timestamp and determine the surface material, allowing for accurate collision sounds between virtual and real objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If virtual audio is presented without accounting for user surroundings and spatial movements, then the system complexity is reduced, but auditory realism and immersion are compromised
Solution Approach 1:
The system performs preliminary actions by capturing audio and video streams before the actual collision event occurs. The microphone captures ambient audio and the sensor captures visual data of the surface material in advance, so that when the virtual object collides with the surface, the system can immediately generate appropriate collision audio without real-time processing delays, thus maintaining both simplicity and realism.
Solution Approach 2:
The system creates a copy of the real environment's acoustic properties by capturing audio and video streams of the actual surface material. This copied information is then used to generate synthetic collision audio that mimics the real surface's acoustic characteristics, allowing the system to achieve high auditory realism without physically replicating the complex acoustic environment.
2Ease of manufacture
If collision audio does not match the surface material and user expectations, then the audio generation process is simplified, but user immersion and comfort are reduced
Solution Approach 1:
The system uses feedback from the captured audio and video streams to generate collision audio that matches the actual surface material and user expectations. The microphone provides feedback about the real acoustic environment, and the sensor provides feedback about the visual appearance of the surface, allowing the system to adjust the generated audio to match both, thereby eliminating auditory mismatches and improving user comfort.
3Device complexity
If virtual audio is not spatialized according to virtual object movement, then the audio processing complexity is reduced, but spatial accuracy and immersion are compromised
Solution Approach 1:
The system applies dynamics by continuously adjusting the spatial positioning of collision audio based on the virtual object's movement and the user's head orientation. The audio is dynamically remixed according to the current spatial relationship between the virtual object, the user, and the captured surface material, ensuring high spatial accuracy without requiring complex pre-computed audio maps.
Data Source
AI summary
Disclosed herein are systems and methods for presenting an audio signal associated with presentation of a virtual object colliding with a surface. The virtual object and the surface may be associated with a mixed reality environment. Generation of the audio signal may be based on at least one of an audio stream from a microphone and a video stream form a sensor. In some embodiments, the collision between the virtual object and the surface is associated with a footstep on the surface.