Mobile Device Media Identification via Audio Video Fingerprinting
Solution Overview
Problem
Existing technologies face challenges in accurately identifying television or media content on mobile devices due to ambient noise and image distortions, which degrades the quality of audio and video signals.
Innovation Solution
A method for mobile devices to capture and process audio and video content from TV displays, using audio fingerprinting techniques such as FFT, filter banks, DCT, and multi-tap FIR filters, along with video contour detection, to generate query signatures for real-time content identification and trigger synchronized actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If audio and video content is captured from TV displays using mobile devices, then real-time content identification capability is enabled, but ambient noise and image distortions degrade signal quality and reduce identification accuracy
Solution Approach 1:
The patent segments the audio signal into multiple frequency bands using filter banks, and segments the video signal into multiple frames and regions. This segmentation allows selective processing and comparison of different portions of the signals, enabling robust identification even when some portions are degraded by noise or distortion.
Solution Approach 2:
The patent transforms audio signals from time domain to frequency domain using FFT, and applies multiple mathematical transformations (filter bank, DCT, log/square root functions) to extract features that are invariant to noise and distortion. Video parameters are also transformed through contour detection and feature extraction to create noise-resistant representations.
2Measurement precision
If complex audio fingerprinting techniques (FFT, filter banks, DCT, multi-tap FIR filters) are applied to extract content features, then identification accuracy is improved, but processing complexity and computational requirements increase
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing reference fingerprints for known content in a database. During real-time identification, the system only needs to compute fingerprints for the captured signal and compare against pre-computed references, significantly reducing processing complexity while maintaining high accuracy.
Solution Approach 2:
The patent extracts only the most salient features from the audio and video signals using selective filtering and feature extraction techniques. By taking out only the essential identifying characteristics rather than processing entire signals, the system achieves high accuracy with reduced computational complexity.
3Reliability
If multiple transformations (FFT, filter bank, DCT, log/square root functions) are applied to audio frames, then robust feature extraction is achieved, but processing time and computational load increase
Solution Approach 1:
The patent merges multiple transformation operations into a unified fingerprint extraction pipeline. By combining FFT, filter bank, DCT, and log/square root functions into a single integrated process that operates on audio frames in sequence, the system achieves robust feature extraction while optimizing processing time through efficient data flow and intermediate result reuse.
Data Source
AI summary
A mobile device responds in real time to media content presented on a media device, such as a television. The mobile device captures temporal fragments of audio-video content on its microphone, camera, or both and generates corresponding audio-video query fingerprints. The query fingerprints are transmitted to a search server located remotely or used with a search function on the mobile device for content search and identification. Audio features are extracted and audio signal global onset detection is used for input audio frame alignment. Additional audio feature signatures are generated from local audio frame onsets, audio frame frequency domain entropy, and maximum change in the spectral coefficients. Video frames are analyzed to find a television screen in the frames, and a detected active television quadrilateral is used to generate video fingerprints to be combined with audio fingerprints for more reliable content identification.