Food Image Captioning Using Contrastive Learning for Ingredient Accuracy

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current food image analysis technologies suffer from low accuracy in identifying food types and ingredients, leading to errors in calorie counting and nutrient analysis, and conventional image captioning methods are inefficient and costly with high labeling requirements.

Innovation Solution

A method and apparatus using image captioning to generate food names by extracting features from food images, inferring ingredients and recipes, and generating accurate captions through a multi-stage encoder-decoder process with contrastive learning to enhance embedding similarity.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional image search is used for food identification, then the process is simple, but the search accuracy is considerably low

Engineering Contradiction:

Improvefood identification accuracyVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent replaces conventional mechanical image search systems with a deep learning-based system that uses convolutional neural networks (CNNs) and transformers to automatically identify food items and ingredients from images, achieving high accuracy without manual intervention

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent changes the fundamental parameters of image processing by using advanced deep learning models that can extract complex features from images, enabling the system to accurately identify food items, ingredients, and even cooking methods based on visual data

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If deep learning is applied to image identification, then classification accuracy improves for learned images, but accuracy for new images not learned before remains low

Engineering Contradiction:

Improveclassification accuracyVSAvoidability to classify new images

Core Design Contradiction:

Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The patent employs dynamic adaptation mechanisms where the deep learning model can adjust its feature extraction and classification capabilities based on the input image, allowing it to handle new and unseen food items effectively through continuous learning and flexible pattern recognition

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system performs preliminary feature extraction and representation learning that enables it to generalize to new images. By pre-training on diverse food datasets and learning universal food representations, the model is prepared to accurately classify new images without requiring retraining

Inventive Principle:

Principle #10Preliminary action

3Loss of information

If conventional image captioning methods are used, then multiple captions can be generated, but labeling cost becomes high and it takes considerable work

Engineering Contradiction:

Improveinformation completenessVSAvoidlabeling time

Core Design Contradiction:

Loss of informationVSLoss of time

Solution Approach 1:

The patent implements a self-service approach where the deep learning system automatically generates accurate food names and ingredient lists from images without requiring human annotators. The model autonomously performs the captioning task that would otherwise require significant human time and effort

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual labeling processes with an automated deep learning system that uses transformers and attention mechanisms to generate accurate food-related text descriptions, eliminating the need for human intervention in the labeling process

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

4Loss of information

If multiple captions are inserted into one image as labels, then more information is provided, but the system becomes less efficient and more costly

Engineering Contradiction:

Improveinformation completenessVSAvoidprocessing efficiency

Core Design Contradiction:

Loss of informationVSProductivity

Solution Approach 1:

The patent merges multiple pieces of information (food name, ingredients, cooking method) into a single integrated caption generation process. The transformer model processes all this information simultaneously and outputs a comprehensive description that combines multiple labels into one efficient output, improving processing efficiency while maintaining information completeness

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS12394222B2Methods and apparatuses for analyzing food using image captioning

Publication Date: 2025.08.19 NUVI LABS CO LTD

AI summary

Provided are a method and an apparatus for analyzing food using image captioning. A method for analyzing food using image captioning according to one embodiment of the present disclosure comprises generating image captioning data using food image features extracted from a food image; and generating a food name for the food image using the generated image captioning data.