Gaze-Responsive Audio-Visual Rendering for Sound Image Localization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies for controlling viewing and audio processing in free viewpoint video systems do not effectively utilize user gaze points to enhance content interaction and engagement.
Innovation Solution
An information processing device that discriminates a user's gazing point to estimate sound image coordinates, performs video rendering based on this point, and outputs related information, using machine learning models for enhanced video and audio processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional video display control is used without gaze detection, then the system is simple and easy to operate, but user engagement and content interaction are not enhanced
Solution Approach 1:
The patent introduces a gaze detection system as an intermediary between the user and the video display system. The detection unit captures user gaze information, which then influences video output control through framing and zooming adjustments. This intermediary mechanism enables adaptive content interaction without requiring complex direct user control interfaces.
Solution Approach 2:
The system implements feedback by continuously monitoring user gaze points and dynamically adjusting video display parameters accordingly. The detection unit provides real-time gaze information that feeds back to the video output control unit, creating a closed-loop system that adapts to user attention and enhances engagement through responsive display adjustments.
2Productivity
If gaze detection and dynamic video rendering are implemented, then user engagement is improved, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary actions by pre-calculating and preparing multiple framed video segments corresponding to different gaze regions. When user gaze is detected, the system can quickly switch between pre-prepared segments rather than performing complex rendering operations in real-time, thereby reducing processing time while maintaining adaptive content delivery.
3Reliability
If only basic video display is used, then the device complexity is low, but audio-visual synchronization and sound image localization are not optimized
Solution Approach 1:
The patent applies local quality by optimizing audio-visual synchronization and sound image localization specifically for the region of user gaze. Instead of uniformly processing the entire video frame, the system focuses computational resources on the gazed region, adjusting framing, zooming, and audio localization parameters locally to enhance synchronization reliability where the user is actually attending.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided is an information processing device that performs processing on a content. An information processing device is provided with an estimation unit that estimates sounding coordinates at which a sound image is generated on the basis of a video stream and an audio stream, a video output control unit that controls an output of the video stream, and an audio output control unit that controls an output of the audio stream so as to generate the sound image at the sounding coordinates. A discrimination unit that discriminates a gazing point of a user who views video and audio is further provided, in which the estimation unit estimates the sounding coordinates at which the sound image of the object gazed by the user is generated on the basis of a discrimination result.