360 VR Viewport Prediction Using Sound Source Localization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current 360 virtual reality (VR) content technologies lack the ability to accurately predict a user's viewport location, particularly in relation to sound sources, leading to higher bandwidth consumption and reduced immersion in virtual reality experiences.
Innovation Solution
A system and method that utilize location information of a sound source in 360 VR content by spatially dividing panorama videos into segment tiles, incorporating sound localization information, and dynamically allocating weights to region of interest, sound source localization, current viewport, and pole items based on bandwidth conditions to determine an adaptive segment bitrate for improved viewport prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If 360 VR content is transmitted with high image quality for all tiles, then user immersion is improved, but bandwidth consumption increases significantly
Solution Approach 1:
The patent applies local quality by transmitting different quality levels for different tile regions based on their importance. Specifically, tiles containing sound sources, viewport regions, and pole regions are transmitted with high quality, while other background tiles are transmitted with lower quality. This differential quality approach maintains user immersion in critical areas while reducing overall bandwidth consumption.
2Measurement precision
If viewport prediction accuracy is improved by considering sound source location, then user satisfaction is enhanced, but system complexity increases
Solution Approach 1:
The patent merges viewport prediction with sound source location information. The client device determines sound source locations from audio data and integrates this information with visual viewport prediction algorithms. By combining audio and visual data streams, the system achieves more accurate viewport prediction without requiring separate complex systems for each modality.
Solution Approach 2:
The system implements feedback by continuously monitoring sound source locations and using this information to adjust viewport prediction and tile quality allocation in real-time. The client device receives audio data, determines sound source positions, and feeds this information back into the viewport prediction and bitrate determination processes to dynamically adapt content transmission.
Data Source
AI summary
There is disclosed a system and method of predicting a location of a user's viewport using location information of a sound source in 360 virtual reality (VR) content. According to the present invention, the spatial representation description (SRD) and the sound source localization information (SLI) of the audio MPD are transmitted with the video MPD being extended, the SRD of the sound source is used to derive the location of the sound source, and the weights of the respective items of the region of interest (ROI), the sound source localization information (SLI), the current viewport, and the pole are allocated differentially according to the bandwidth condition and then multiplied by the reference bitrate to determine the segment bitrate to be provided next, thereby improving the accuracy of predicting the location of the user's viewport and ensuring the quality of experience (QoE) for regeneration speed and regeneration quality.