Mobile Device Media Identification via Audio Video Fingerprinting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in reliably identifying television or media programming content on mobile devices due to ambient noise and image distortions, which degrades the accuracy of audio and video signal processing.
Innovation Solution
A method for mobile devices to capture and process audio and video signals from a television, using contour detection, audio fingerprinting techniques like FFT and DCT, and multi-dimensional feature extraction to generate query signatures for real-time content identification and synchronization with media applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If audio and video signals are captured from television in ambient environments, then real-time content identification capability is enabled, but measurement precision deteriorates due to ambient noise and image distortions
Solution Approach 1:
The patent segments the content identification process into multiple independent modules: audio fingerprinting module, video fingerprinting module, and combined identification module. Each module processes specific signal types separately, allowing optimized processing for each modality while maintaining overall system accuracy despite ambient interference
Solution Approach 2:
The patent introduces fingerprint templates as intermediary representations that capture essential content characteristics while filtering out ambient noise and distortions. These fingerprints serve as mediators between raw captured signals and final identification results, enabling accurate matching even in noisy environments
2Reliability
If multiple signal processing operations are performed for robust identification, then reliability improves, but device complexity increases
Solution Approach 1:
The patent implements dynamic processing where the system adaptively selects processing operations based on input quality and identification needs. The fingerprint extraction and matching processes dynamically adjust their complexity, performing simpler operations when signals are clear and more complex operations when ambient interference is detected, thereby maintaining reliability while managing device complexity
Solution Approach 2:
The patent merges audio and video identification results through a unified matching algorithm that combines fingerprints from both modalities. This combination approach enhances reliability by cross-validating results across different signal types while using a single integrated processing framework that manages overall system complexity
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables accurate and real-time identification of television content on mobile devices, even in noisy environments, by generating query signatures that can efficiently search multimedia databases for matching content, triggering actions synchronized with the identified media.
Implementation Method 1
Video content displayed on the TV display device is captured as a set of video frames from an optical image capture device on the mobile device
Implementation Method 2
Audio content generated on a television (TV) display device is captured as a waveform from a microphone on the mobile device
Implementation Method 3
A fast Fourier transform (FFT) for the windowed audio frame samples is computed which FFT results are filtered by a filter bank on the spectral components
Implementation Method 4
A discrete cosine transform (DCT) is applied separately to the log and square root output coefficients to produce two sets of DCT spectral components
Data Source
AI summary
A mobile device responds in real time to media content presented on a media device, such as a television. The mobile device captures temporal fragments of audio-video content on its microphone, camera, or both and generates corresponding audio-video query fingerprints. The query fingerprints are transmitted to a search server located remotely or used with a search function on the mobile device for content search and identification. Audio features are extracted and audio signal global onset detection is used for input audio frame alignment. Additional audio feature signatures are generated from local audio frame onsets, audio frame frequency domain entropy, and maximum change in the spectral coefficients. Video frames are analyzed to find a television screen in the frames, and a detected active television quadrilateral is used to generate video fingerprints to be combined with audio fingerprints for more reliable content identification.