Aligning 2D Visual Objects with Volumetric Audio in VR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current VR/AR/MR applications face challenges in aligning 2D visual content with 6DoF audio content, leading to misalignment of audio sources and visual counterparts, which affects the immersive experience.
Innovation Solution
A method and apparatus that determine and align the spatial position and orientation of objects of interest in 2D visual content with related audio sources in a presentation volume, ensuring that the 2D visual content and volumetric audio content are rendered in synchronization, allowing users to experience audio and visual elements from a uniform direction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 2D visual content and 6DoF audio content are presented independently without alignment, then device complexity is reduced, but audio-visual alignment precision deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-determining the spatial positions and orientations of objects in 2D visual content before rendering. The system identifies objects of interest, determines their spatial coordinates in advance, and prepares alignment data structures (such as bounding boxes and transformation matrices) before the actual rendering process, ensuring audio-visual synchronization without adding complex real-time processing
Solution Approach 2:
The patent introduces an intermediary alignment mechanism that acts as a mediator between 2D visual content and 6DoF audio content. This intermediary system uses spatial coordinate transformation and object identification to bridge the two different content types, allowing them to be synchronized without directly complicating the core rendering pipelines of either content type
2Reliability
If spatial alignment processing is implemented to synchronize audio and visual content, then audio-visual coherence is improved, but processing time increases
Solution Approach 1:
The patent applies self-service by enabling the 2D visual content to carry its own spatial information metadata that describes the positions and orientations of objects within the visual content. This self-contained spatial data allows the system to perform alignment operations using information already present in the visual content structure, reducing the need for external processing and minimizing additional processing time
Solution Approach 2:
The patent utilizes parameter changes by transforming spatial coordinates and orientation parameters of objects from the 2D visual content into the 6DoF audio spatial reference frame. By changing the parameter representation (coordinates, angles, positions) rather than the fundamental content structure, the system achieves alignment efficiently through mathematical transformation rather than complex reprocessing
Data Source
Figure 1A~1C
Figure 2
Figure 3
AI summary
There is provided rendering 2D visual content and related volumetric audio content. A method comprises determining at least one object of interest of two-dimensional, 2D, visual content related to an audio source of volumetric audio content, aligning a spatial position and orientation of the at least one object of interest with a spatial position and orientation of the related audio source in a presentation volume, rendering the 2D visual content and the volumetric audio content on the basis of the aligned spatial position and orientations of the at least one object of interest and the related audio source.