Directional Audio Processing for Immersive Scene Stabilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing audio codecs lack the ability to efficiently handle spatial rotations and translations of captured audio scenes due to device movements, leading to potential motion sickness and instability in immersive audio experiences.
Innovation Solution
A device at the capturing end processes directional audio by modifying it based on spatial metadata, encoding this modified audio, and optionally including partial spatial data in the bitstream, allowing for compensation of device movements without requiring full metadata transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If full spatial metadata is transmitted to compensate for device movements, then audio scene stability is improved, but data transmission bitrate increases
Solution Approach 1:
The patent extracts and transmits only the essential spatial transformation parameters (rotation and translation data) from the complete spatial metadata, rather than transmitting all spatial information. This selective extraction maintains audio scene stability while reducing the bitrate requirement.
Solution Approach 2:
The patent applies partial action by transmitting only the necessary portion of spatial data required for motion compensation. The system determines the minimal set of spatial parameters needed to stabilize the audio scene during device movement, avoiding transmission of redundant information.
2Object-affected harmful factors
If spatial rotation compensation is applied during audio capture, then motion sickness is reduced, but processing complexity increases
Solution Approach 1:
The patent applies preliminary action by performing spatial rotation compensation during the audio capture phase rather than during rendering. The capturing device pre-processes the directional audio data to compensate for its own movements, eliminating motion sickness causes before the audio is transmitted or rendered.
Solution Approach 2:
The capturing device performs self-service by using its own spatial sensors (accelerometers, gyroscopes) to detect its movement and automatically compensating for these movements in the captured audio data. The device serves its own motion compensation needs without requiring external processing.
3Adaptability or versatility
If directional audio processing is performed at the capturing device, then rendering flexibility is improved, but capturing device computational load increases
Solution Approach 1:
The patent segments the audio processing tasks between the capturing device and rendering devices. The capturing device performs only the essential directional audio encoding and spatial parameter extraction, while the rendering flexibility is achieved at the receiving end where multiple rendering configurations can be applied to the encoded data.
Data Source
Figure 1~2
Figure 3~5
Figure 6~7
AI summary
The disclosure herein generally relates to capturing, acoustic preprocessing, encoding, decoding, and rendering of directional audio of an audio scene. In particular, it relates to a device adapted to modify a directional property of a captured directional audio in response to spatial data of a microphone system capturing the directional audio. The disclosure further relates to a rendering device configured to modify a directional property of a received directional audio in response to received spatial data.