Joint Embedding Model for Visual Concept Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for searching visual-media items in social-networking systems face challenges in identifying and classifying visual concepts within images, as they rely on limited and noisy text sources, and struggle with evolving slang terms and new visual concepts, leading to inefficient search results.
Innovation Solution
The system employs an image-recognition process to segment images and associate visual concepts with n-grams from user communications, using a joint embedding model to leverage crowdsourced information and train a visual-concept recognition system, allowing for efficient indexing and retrieval of visual concepts across languages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If current methods use limited text sources (title, description, comments) to identify visual concepts, then the search system is simple to operate, but the measurement precision of visual concept identification deteriorates
Solution Approach 1:
The patent merges multiple text sources (title, description, comments, captions, hashtags) into a unified text pool for training the image-recognition system. This combination of previously separate text sources improves visual concept identification accuracy while maintaining system simplicity through automated processing.
Solution Approach 2:
The system automatically trains the image-recognition model using crowdsourced text data from user communications without requiring manual annotation. The system serves itself by leveraging user-generated content to improve its own visual concept identification capabilities.
2Device complexity
If the system uses a fixed set of keywords for visual concepts, then the device complexity is reduced, but the adaptability to evolving slang terms and new visual concepts deteriorates
Solution Approach 1:
The patent implements a dynamic keyword system where the image-recognition model is continuously trained on new user communications to adapt to evolving slang terms and emerging visual concepts. The system transitions from static keywords to dynamically updating visual concept representations.
Solution Approach 2:
The system performs preliminary training on historical user communications to build an initial visual concept dictionary before deployment. This pre-training enables the system to handle common slang and concepts from the outset, reducing the need for complex real-time adaptations.
3Measurement precision
If the system processes language-specific information for each language, then the measurement precision of visual concept recognition is improved, but the device complexity increases
Solution Approach 1:
The patent implements a universal image-recognition model that processes multiple languages simultaneously through a single training framework. The model learns visual-concept associations from multilingual user communications without requiring separate processing pipelines for each language, reducing system complexity while maintaining accuracy.
Data Source
AI summary
In one embodiment, a method includes identifying a shared visual concept in visual-media items based on shared visual features in images of the visual-media items; extracting, for each of the visual-media items, n-grams from communications associated with the visual-media item; generating, in a d-dimensional space, an embedding for each of the visual-media items at a location based on the visual concepts included in the visual-media item; generating, in the d-dimensional space, an embedding for each of the extracted n-grams at a location based on a frequency of occurrence of the n-gram in the communications associated with the visual-media items; and associating, with the shared visual concept, the extracted n-grams that have embeddings within a threshold area of the embeddings for the identified visual-media items.