Multimodal Food Identification for Unseen Image Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current food identification technologies using image classification models struggle with low accuracy, especially for new or unseen food images, due to their reliance on pre-learned data, leading to errors in calorie counting and food type recognition.
Innovation Solution
A multimodal model is pre-learned using a plurality of food images and associated texts, such as names, recipes, and descriptions, to enhance similarity between image and text features, allowing for accurate identification of unknown food images by grouping and separating indexes of different classes during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a typical image classification model is used for food identification, then the classification accuracy for previously learned images is improved, but the classification accuracy for new unseen images deteriorates
Solution Approach 1:
The patent transitions from single-modal image classification to multi-modal classification by incorporating both image features and text features. The classification model processes images and corresponding text descriptions through separate encoding pathways, then fuses the features in a shared feature space to enable accurate classification of both seen and unseen food images
Solution Approach 2:
The patent implements a pre-learning stage before the main classification task. During pre-learning, the model learns to align image features with text features by minimizing the distance between matching pairs and maximizing the distance between non-matching pairs. This preliminary alignment enables the model to generalize to new food images during inference
Data Source
AI summary
Provided are a method and an apparatus for identifying food using a multimodal model. A method for identifying food using a multimodal model according to one embodiment of the present disclosure comprises setting a plurality of food images belonging to different classes as pre-learning targets; pre-learning a multimodal model for each of the plurality of food images set as the pre-learning target so that a text feature paired with an image feature is more similar to the image feature than the other text features not paired with the image feature; and identifying food to be identified from input food images using the pre-learned multimodal model.