Joint Image-Audio Search System for Relevance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image search systems rely solely on visual similarity, which can lead to irrelevant results as users interpret images subjectively, resulting in searches returning images of different brands or types when interested in specific categories, such as cars.
Innovation Solution
A joint image-audio search system that processes queries including both image and audio data, where the audio provides additional context to improve relevance by identifying the object of interest within the image and supplementing user search parameters, using a trained joint image-audio relevance model to generate scores for resources based on both visual and textual features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If image search is based only on visual similarity, then the search process is simple and fast, but the search results are not relevant to user intent
Solution Approach 1:
The patent combines image search and audio search into a unified joint image-audio search system. The image processing component extracts visual features from query images, while the audio processing component extracts features from query audio. Both feature sets are integrated and processed together by the relevance model to generate comprehensive search results that satisfy both visual and auditory query requirements, thereby improving search relevance without excessive complexity increase
Solution Approach 2:
The joint image-audio relevance model serves multiple functions: it processes image features, audio features, and their combinations; it handles both exact match and similarity search; it provides relevance scoring for diverse resource types including images, audio files, and web pages. This multi-functionality allows the system to improve reliability across different search scenarios while maintaining a unified architecture
2Reliability
If audio data is added to image query, then search relevance is improved, but processing complexity increases
Solution Approach 1:
The query processing is segmented into distinct components: image processing module that extracts visual features, audio processing module that extracts audio features, and a relevance modeling module that integrates both feature sets. Each component handles specific tasks independently, then combines results systematically. This segmentation manages processing complexity by breaking down the joint image-audio processing into manageable, specialized sub-tasks
Solution Approach 2:
The patent introduces feature extraction intermediaries that convert raw image and audio data into standardized feature representations before feeding them to the relevance model. These intermediaries act as mediators that normalize different data types (visual and auditory) into a common feature space, facilitating their integration without directly coupling the image and audio processing pipelines, thus managing complexity
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing joint image-audio queries. In one aspect, a method includes receiving, from a client device, a joint image-audio query including query image data and query audio data. Query image feature data is determined from the query image data. Query audio feature data is determined from the audio data. The query image feature data and the query audio feature data are provided to a joint image-audio relevance model trained to generate relevance scores for a plurality of resources, each resource including resource image data defining a resource image for the resource and text data defining resource text for the resource. Each relevance score is a measure of the relevance of corresponding resource to the joint image-audio query. Data defining search results indicating the order of the resources is provided to the client device.