Image-Based Item Recommendations Through Image-to-Text Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing e-commerce recommendation systems rely heavily on textual information, which is not always available or convenient for customers, making it challenging to provide accurate item recommendations based on images.
Innovation Solution
A system that utilizes a multimodal pipeline to convert anchor images into textual information, generate queries, and employ language and machine learning models to provide item recommendations, including similar, complementary, and complete-the-look suggestions, with filtering processes to ensure appropriateness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If recommendation systems rely on textual information, then existing systems can process customer inputs, but accurate recommendations cannot be provided when textual information is unavailable or insufficient
Solution Approach 1:
The patent introduces an image-to-text conversion module as an intermediary that bridges the gap between image inputs and textual processing. This module converts anchor images into textual descriptions that can be processed by language models, enabling the system to handle cases where direct textual information is unavailable while maintaining accurate recommendations through multi-modal data fusion
Solution Approach 2:
The system achieves multi-functionality by accepting both textual inputs and image inputs through a unified recommendation framework. The language model can process either customer-provided text or AI-generated text from image conversion, making the system universally applicable to various input modalities and improving reliability across different scenarios
2Ease of operation
If customers provide textual information manually, then specific recommendation needs can be expressed, but time and convenience are consumed
Solution Approach 1:
The system implements self-service by automatically converting anchor images into textual descriptions using image-to-text conversion models. This eliminates the need for customers to manually provide textual information, as the system autonomously extracts and processes information from images, significantly reducing customer input time and effort while maintaining recommendation accuracy
Solution Approach 2:
The image-to-text conversion is performed preliminarily before the recommendation generation process. By pre-converting images to text representations, the system prepares all necessary textual data in advance, allowing the language model to generate recommendations efficiently without requiring real-time customer input
3Reliability
If image-based recommendations are implemented, then accurate recommendations can be provided without textual information, but system complexity increases
Solution Approach 1:
The patent segments the complex image-based recommendation system into distinct modular components: image-to-text conversion module, language model module, and recommendation generation module. This segmentation allows each component to perform its specific function independently, making the overall complex system more manageable and easier to implement while maintaining high recommendation accuracy through coordinated operation of these modules
Data Source
AI summary
Systems and methods for providing item recommendations based on item images or uploaded images are disclosed. In some embodiments, a disclosed method includes: receiving, from a computing device, a recommendation request for recommending items to a customer; determining an anchor image based on the recommendation request; generating at least one query based on the anchor image; generating, using a language model, textual recommendation data based on the at least one query; generating, using at least one machine learning model, at least one ranked list of recommended items based on the textual recommendation data; and transmitting to the computing device the at least one ranked list of recommended items to be displayed to the customer.