Audio Source Tracking via Visual Object Association
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional audio tracking techniques are inaccurate due to echoes, noise, and directional microphone characteristics, making it difficult to track audio sources, especially when they are silent or move outside the microphone's sensitive region, leading to ambiguity in determining whether a sound source has moved or a new source is emitting sound.
Innovation Solution
A system and method that processes video and audio signals to identify and match visual objects with audio sources, using geometric modeling and computer vision to track audio sources indirectly by tracking visual objects, and synthesizes audio for a virtual microphone position, simulating the sound that would have been recorded by an actual microphone at that position.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional audio tracking techniques are used, then the system is simple to implement, but the tracking accuracy deteriorates due to echoes, noise, and directional microphone characteristics
Solution Approach 1:
The patent introduces visual objects as an intermediary to track audio sources. Instead of directly tracking audio signals which are affected by echoes and noise, the system tracks visual objects (video frames) that are associated with audio sources. This visual intermediary provides accurate position information without being affected by acoustic interference, thereby improving tracking accuracy while maintaining reasonable system complexity.
Solution Approach 2:
The patent replaces the acoustic-based tracking system with a vision-based tracking system. By substituting the mechanical/acoustic measurement method (microphone-based audio tracking) with an optical method (video-based visual object tracking), the system achieves higher accuracy since visual tracking is not affected by echoes, noise, or directional microphone characteristics.
2Reliability
If directional microphones are used, then the audio recording is directional and focused, but the tracking accuracy deteriorates when audio sources move outside the sensitive region or when microphones introduce coloration and delays
Solution Approach 1:
The patent makes the tracking system universal by using visual object tracking that works regardless of audio source position or microphone characteristics. The visual tracking method can track objects anywhere in the field of view, providing universal coverage that adapts to any position, eliminating the limited sensitive region problem of directional microphones.
Solution Approach 2:
The patent changes the measurement parameter from acoustic properties (affected by microphone directionality and position) to visual properties (position of visual objects in video frames). This parameter change allows tracking to work reliably across the entire visual field rather than being constrained to a narrow acoustic sensitive region.
3Loss of information
If audio sources are tracked directly using audio signals, then the tracking method is straightforward, but the system cannot track silent audio sources or distinguish between moved and new sources
Solution Approach 1:
The patent performs preliminary association between visual objects and audio sources when they are both detectable. By establishing this association in advance (when the audio source is emitting sound and visible), the system creates a persistent link that allows continuous tracking of the visual object even when the audio source becomes silent or moves out of audio detection range.
Solution Approach 2:
The patent uses visual objects as an intermediary to maintain continuous tracking information. When audio sources are silent or ambiguous, the visual object serves as a mediator that preserves the tracking data, allowing the system to distinguish between a moved source and a new source by comparing visual object identities rather than relying solely on audio signal presence.
Data Source
AI summary
Disclosed is a system and method for generating a model of the geometric relationships between various audio sources recorded by a multi-camera system. The spatial audio scene module associates source signals, extracted from recorded audio, of audio sources to visual objects identified in videos recorded by one or more cameras. This association may be based on estimated positions of the audio sources based on relative signal gains and delays of the source signal received at each microphone. The estimated positions of audio sources are tracked indirectly by tracking the associated visual objects with computer vision. A virtual microphone module may receive a position for a virtual microphone and synthesize a signal corresponding to the virtual microphone position based on the estimated positions of the audio sources.