Food Image Analysis via Sequential Captioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies for analyzing food eating habits face challenges due to low accuracy in food type identification from images, leading to errors in calorie counting and high labeling costs in image captioning methods.
Innovation Solution
A method and apparatus using image captioning to generate accurate food names from food images by extracting features and sequentially generating captions for food ingredients, recipes, and names, employing pre-learned encoders and decoders through contrastive learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional image search technology is used for food identification, then the system is simple to operate, but the search accuracy and food type identification accuracy are considerably low
Solution Approach 1:
The patent segments the food identification process into multiple stages: initial image search, image captioning generation, and iterative refinement. Each stage processes specific aspects of the food image independently, allowing the system to improve accuracy without requiring complete system redesign. The segmentation enables progressive refinement from basic identification to detailed food type classification.
Solution Approach 2:
The patent applies preliminary action by generating image captions and food ingredient information before final food type identification. The system pre-processes the image data through captioning models and ingredient recognition models, creating enriched feature representations that improve subsequent identification accuracy. This preliminary processing prepares the data in advance for more accurate classification.
2Measurement precision
If deep learning classification models are used for food identification, then previously learned images can be classified with high accuracy, but new images not learned before have lower classification accuracy
Solution Approach 1:
The patent implements dynamics by making the food identification system adaptive and evolving. The iterative refinement process allows the system to dynamically adjust its predictions based on generated captions and ingredient information. The system transitions from static pre-trained classification to a dynamic process that continuously improves accuracy by incorporating new information from multiple models and refinement iterations.
Solution Approach 2:
The patent introduces intermediary elements between the input image and final classification: image captioning models generate descriptive text, ingredient recognition models identify components, and these intermediaries bridge the gap between raw images and classification. These intermediary representations enable the system to handle new food images by decomposing them into recognizable components and descriptions.
3Measurement precision
If conventional image captioning methods are applied to food images, then multiple captions can be generated per image, but the labeling cost becomes high and it takes considerable work to write multiple captions
Solution Approach 1:
The patent applies self-service by enabling the system to automatically generate its own labels and captions through integrated AI models. The image captioning model, ingredient recognition model, and food type identification model work together in an automated pipeline, eliminating the need for manual caption writing. The system serves itself by generating descriptive information and using it for identification without human intervention.
Solution Approach 2:
The patent merges multiple functions into a unified system: image captioning, ingredient recognition, and food type identification are combined into an integrated workflow. The generated captions and ingredient information are immediately utilized by the identification model, eliminating separate manual labeling steps. This merging of functions improves productivity by processing multiple tasks in a single automated pipeline.
4Loss of information
If multiple captions are generated per food image, then comprehensive description is achieved, but similar-looking food images produce the same caption multiple times causing adverse effects in learning AI models
Solution Approach 1:
The patent implements feedback mechanisms where the generated captions and ingredient information are fed back into the identification process. The system uses the captioning output as input for ingredient recognition, which in turn informs food type identification. This feedback loop ensures that redundant captions are filtered through multiple processing stages, with each stage refining the information and eliminating duplicates before final classification.
Solution Approach 2:
The patent applies local quality by focusing the captioning and identification process on specific local features of food images: ingredient components, cooking methods, and dish characteristics. Rather than generating generic repeated captions, the system locally analyzes different aspects of the image (ingredients, preparation style, presentation) to generate diverse and informative descriptions that differentiate similar-looking foods.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided are a method and an apparatus for analyzing food using image captioning. A method for analyzing food using image captioning according to one embodiment of the present disclosure comprises generating image captioning data using food image features extracted from a food image; and generating a food name for the food image using the generated image captioning data.