Text Embedding via Visual Grounding and Gating Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text processing systems using neural networks face challenges in generating effective embeddings for large vocabularies without excessive computational resources and in improving the performance of machine learning tasks such as translation and natural language processing.
Innovation Solution
A system that generates numeric embeddings of text segments by leveraging a pre-trained convolutional neural network and an image search engine, incorporating visual features to create visually-grounded embeddings, and combines these with other embeddings using a gating neural network for task-specific weight determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional embeddings are used for large vocabularies, then the system can process text, but the performance of machine learning tasks is insufficient
Solution Approach 1:
The patent combines multiple embedding sources (image search results and alternative embedding sources) to create enriched text embeddings. The system merges visual embeddings from image search with traditional text embeddings, allowing the model to leverage both visual and textual information for improved performance on machine learning tasks.
Solution Approach 2:
The patent introduces an intermediary embedding generation process that uses image search results as a bridge between text input and machine learning tasks. The system generates embeddings from image search results and combines them with alternative embeddings, creating an intermediate representation that enhances the original text embedding quality.
2Measurement precision
If computationally intensive embedding generation is used, then embedding quality improves, but computational resources are excessive
Solution Approach 1:
The patent performs preliminary embedding generation for image search results and stores these embeddings for later use. By pre-computing and caching the embeddings from image search results, the system avoids the need to re-generate these computationally intensive embeddings during each machine learning task, thus reducing overall computational resource consumption while maintaining high embedding quality.
Solution Approach 2:
The patent uses partial embedding generation by leveraging pre-existing image search results and their embeddings rather than generating all possible embeddings from scratch. The system selectively generates and combines only the necessary embeddings for each task, avoiding excessive computational resources by using partial information from image search and alternative sources.
3Adaptability or versatility
If a large vocabulary is processed, then text coverage is improved, but computational complexity increases
Solution Approach 1:
The patent creates a universal embedding generation approach that works across a large vocabulary by leveraging image search results and alternative embedding sources. The system uses a multi-functional embedding combination mechanism that can handle diverse text segments uniformly, allowing it to process large vocabularies without proportionally increasing computational complexity through consistent embedding generation and combination procedures.
Data Source
AI summary
Methods, systems, and apparatus including computer programs encoded on a computer storage medium, for generating a data set that associates each text segment in a vocabulary of text segments with a respective numeric embedding. In one aspect, a method includes providing, to an image search engine, a search query that includes the text segment; obtaining image search results that have been classified as being responsive to the search query by the image search engine, wherein each image search result identifies a respective image; for each image search result, processing the image identified by the image search result using a convolutional neural network, wherein the convolutional neural network has been trained to process the image to generate an image numeric embedding for the image; and generating a numeric embedding for the text segment from the image numeric embeddings for the images identified by the image search results.