Image Response Suggestions Using LSTM Feature Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies fail to provide accurate and relevant automatic response suggestions for images in messages, often resulting in incomplete or inappropriate responses, especially when no objects are recognized or when multiple objects are present in an image.
Innovation Solution
A computer-implemented method that detects images in messages, analyzes them to determine a feature vector, and uses a conditioned language model, such as a long-short term memory (LSTM) network, to generate conversational responses by selecting word sequences based on probabilities, allowing for context-specific and varied response suggestions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If object recognition is used to generate response suggestions, then responses can be based on identified content, but the system fails when no objects are recognized or when multiple objects are present
Solution Approach 1:
The patent replaces the mechanical object recognition system with a deep learning-based image embedding system. Instead of identifying specific objects in images, the system uses a pre-trained neural network to generate comprehensive feature vectors that capture the overall semantics and context of the image, enabling reliable response generation across all image types without depending on object detection success
Solution Approach 2:
The patent changes the fundamental parameter representation from discrete object labels to continuous high-dimensional image embeddings. This transformation allows the system to handle diverse image cases uniformly by comparing semantic similarities in the embedding space rather than relying on specific object recognition results
2Adaptability or versatility
If a language model is trained with previous responses to generate suggested responses, then responses can be context-specific and varied, but computational resources are consumed
Solution Approach 1:
The patent performs preliminary action by pre-training the language model offline using large datasets of image-caption pairs and previous user responses. The model learns to map image embeddings to relevant text responses in advance, so that during actual usage, only inference is required rather than full training, significantly reducing online computational resource consumption while maintaining response variety and context-specificity
Data Source
AI summary
Implementations relate to automatic response suggestions to images included in received messages. In some implementations, a computer-implemented method includes detecting an image posted within a first message by a first user, and programmatically analyzing the image to determine a feature vector representative of the image. The method programmatically generates one or more suggested responses to the first message based on the feature vector, each suggested response being a conversational reply to the first message. Generating the suggested responses includes determining probabilities associated with word sequences for the feature vector using a model trained with previous responses to previous images, and selecting one or more of the word sequences based on the associated probabilities. The suggested responses are determined based on the selected word sequences. The method causes the suggested responses to be rendered in the messaging application as one or more suggestions to a second user.