Search Relevance via Text Summaries and Image Captions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search technologies, including prefix and semantic searches, often fail to surface relevant search results due to limitations in text embedding models that exclude content, leading to incomplete or irrelevant search results, especially with longer documents and partially typed queries.
Innovation Solution
A multi-faceted approach combining text summarization, image captioning, and lexical searches to generate comprehensive search results by creating semantic and lexical search data, which includes text summaries and image captions, enabling a more robust search by representing content in a more comprehensive manner.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If text embedding models are used to perform semantic searches, then search results can be identified that are semantically similar to a search query, but relevant content is excluded from being represented in the text embedding due to limited text input length
Solution Approach 1:
The patent segments the content representation into multiple components: a text embedding that captures semantic meaning and an image caption that provides additional context. This segmentation allows each component to serve its specific purpose without being constrained by the limited text input length of embedding models, thereby resolving the contradiction between search relevance and content representation completeness.
Solution Approach 2:
The image caption acts as an intermediary that bridges the gap between the limited text embedding and the full content. By generating a text description from the image, the system creates an additional representation layer that complements the text embedding, enabling more comprehensive content representation without increasing the text input length to the embedding model.
2Measurement precision
If only semantic search approach is used to identify relevant search results, then semantic similarity can be captured, but search results relevant to partial queries are omitted
Solution Approach 1:
The patent merges semantic search and prefix search approaches into a unified system. The semantic search component handles semantic similarity for complete queries, while the prefix search component handles partial queries by identifying literal matches. By combining both approaches, the system achieves both semantic search accuracy and adaptability to partial queries, resolving the contradiction between precision and versatility.
3Productivity
If text embedding models consume limited amount of text to produce embeddings, then processing efficiency is improved, but longer documents are excluded from being represented
Solution Approach 1:
The patent segments the content representation task into two parts: using the text embedding model efficiently for semantic representation and using image captioning to handle the remaining content information. This segmentation allows the embedding model to maintain its efficiency with limited text input while the image captioning component compensates for the lost information in longer documents.
Solution Approach 2:
The patent creates a textual copy of the image content through image captioning. This copy serves as an alternative representation that complements the text embedding, enabling the system to represent longer documents comprehensively without requiring the embedding model to process the entire document text, thus maintaining processing efficiency.
Data Source
AI summary
Methods, computer systems, computer-storage media, and graphical user interfaces are provided for providing comprehensive search results. In embodiments, a search query is obtained, and a query text embedding is generated to represent the search query. The query text embedding is compared to a set of text embeddings representing text summaries generated for corresponding content items having text. Based on the comparing, a content item, of the content items, is identified as semantically similar to the search query. Thereafter, a search result indicating the content item identified as semantically similar to the search query is provided for display.