360° Image Stitching With Spatial Audio Blind Spot Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 360-degree cameras suffer from blind spots due to non-overlapping camera fields of view, leading to discrepancies between visual and audio directions, particularly for nearby objects, which can ruin immersion in virtual reality experiences and affect industrial analysis and ASMR videos.
Innovation Solution
A method to modify spatial audio data by adjusting audio features based on distance and direction from the camera, and a method to modify visual data by filling gaps with artificial imagery in the blind spot areas using machine learning or database data, ensuring accurate alignment of sound and visual directions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Area of stationary object
If 360-degree cameras are used to capture visual content all around, then the field of view coverage is improved, but blind spots appear in nearby areas due to non-overlapping camera fields of view
Solution Approach 1:
The patent uses audio data as an intermediary to identify and fill visual blind spots. When a sound is detected in a direction where no visual data exists (blind spot), the system generates synthetic visual content at that location to complement the audio information, thereby mediating between the audio input and the visual output to achieve complete 360-degree coverage without actual blind spots
Solution Approach 2:
The patent segments the 360-degree visual field into multiple regions captured by different cameras, identifies blind spot areas where no camera coverage exists, and separately processes these regions by generating synthetic visual content based on audio data, rather than attempting to capture all areas with physical cameras
2Area of stationary object
If spatial audio data is captured from all directions, then audio coverage is improved, but discrepancies arise between audio and visual directions for nearby objects in blind spot areas
Solution Approach 1:
The system continuously monitors the alignment between audio and visual data, detects discrepancies in direction and distance for nearby objects, and provides feedback by adjusting or generating visual content to match the audio direction, thereby maintaining accurate alignment between audio and visual directions throughout the 360-degree field of view
Solution Approach 2:
The patent performs preliminary analysis of audio data to determine sound source direction and distance before generating visual content, ensuring that the synthetic visual elements are created with correct spatial positioning that matches the audio direction, thus preventing alignment discrepancies rather than correcting them afterward
3Area of stationary object
If multiple cameras are used to reduce blind spots, then visual coverage is improved, but device complexity increases
Solution Approach 1:
The patent creates synthetic copies of visual content in blind spot areas based on audio information rather than using additional physical cameras. This virtual copying approach achieves complete 360-degree visual coverage without increasing the physical camera count, thereby maintaining simple device architecture while eliminating blind spots
Solution Approach 2:
The patent transitions from a two-dimensional camera array problem to a three-dimensional spatial solution by using audio data (which provides directional and distance information in 3D space) to generate visual content, adding a temporal and spatial dimension to the image generation process rather than simply adding more cameras
Data Source
AI summary
There is herein disclosed an apparatus comprising: means for capturing first visual data associated with a first image, means for capturing second visual data associated with a second image, means for capturing spatial audio data from a sound source, means for estimating a first distance of the sound source from the apparatus, means for estimating a direction of the sound source from the apparatus, means for combining at least a portion of the first visual data and at least a portion the second visual data to produce a stitched image by using a transformation parameter, means for modifying the spatial audio data based on the first distance and the transformation parameter to produce modified spatial audio data, and means for outputting the stitched image alongside the modified spatial audio data.