Wearable Audio-Visual Object Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing wearable devices struggle to seamlessly integrate augmented reality (AR) with real-world audio signals, failing to effectively notify users of external events while maintaining an immersive virtual experience.
Innovation Solution
A wearable device equipped with a camera, microphones, and processing circuitry that obtains audio signals and identifies external objects generating sounds, allowing it to display relevant information or images from the camera's view alongside the virtual reality interface.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a wearable device displays a full virtual reality screen occupying the entire display region, then the immersive virtual experience is enhanced, but the user's awareness of external real-world events is reduced
Solution Approach 1:
The display region is segmented into multiple areas: a first screen for virtual reality content and a second screen for real-world information. This segmentation allows both virtual and real-world information to be displayed simultaneously without compromising either the immersive experience or external awareness.
Solution Approach 2:
The patent introduces a spatial dimension to the display by overlaying real-world information in a separate screen region alongside virtual reality content. This dimensional separation enables users to access both virtual and real-world information without switching contexts, resolving the contradiction between immersion and awareness.
2Loss of information
If the wearable device displays real-world information alongside virtual reality content, then external event awareness is improved, but the display complexity and interface management become more difficult
Solution Approach 1:
By dividing the display into distinct first and second screens with specific functions, the system simplifies interface management despite showing multiple types of information. Each screen region has a dedicated purpose, making the overall complex information presentation manageable through clear spatial organization.
3Measurement precision
If the wearable device processes and displays segmented image portions based on audio signal analysis, then the precision of external object identification is improved, but the processing time and computational load increase
Solution Approach 1:
The system performs preliminary audio signal analysis to identify potential sound sources before processing visual information. By pre-identifying objects of interest through audio cues, the system can then focus computational resources on segmenting and displaying only relevant image portions, reducing overall processing time while maintaining high identification precision.
Solution Approach 2:
Instead of processing the entire image, the system selectively segments and processes only specific portions of the image that correspond to identified sound sources. This partial processing approach significantly reduces computational load and processing time while maintaining precise identification of relevant external objects.
Data Source
AI summary
According to an embodiment, a processor of a wearable device obtains, while displaying a first screen with respect to a virtual space on an entire displaying region of a display, an audio signal including sound of an external space by using one or more microphones. The processor identifies, in response to detection of an external object generating the sound by using the audio signal, data corresponding to the external object in information associated with the external space obtained based on the camera. The processor, based on identifying the data in the information, displays, together with the first screen in the displaying region, a second screen including a portion segmented from an image obtained from the camera based on the identified data.