AR Recipe Guidance Using Pose and Object Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users of augmented reality (AR) and mixed reality (MR) devices face challenges in preparing recipes due to difficulties in accessing information, handling ingredient substitutions, and adjusting cooking techniques without manual interaction, leading to potential errors in cooking times and temperatures.
Innovation Solution
A system that utilizes machine-learning algorithms to analyze user actions and predict the recipe being prepared, providing real-time augmented reality suggestions based on detected objects and actions, allowing hands-free interaction and improved cooking guidance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If users manually search for information or interact with touchscreen devices while preparing recipes, then they can access recipe instructions and ingredient information, but they cannot perform hands-free operations and must stop cooking tasks to interact with the device
Solution Approach 1:
The system automatically detects objects in the kitchen environment using image recognition, identifies the recipe being prepared based on detected ingredients and tools, and provides relevant cooking instructions without requiring user initiation. The device serves itself by autonomously gathering and processing information about the cooking context.
Solution Approach 2:
The patent replaces manual touchscreen interaction and voice commands with automated computer vision and machine learning algorithms. The system uses image recognition to detect objects and infer cooking context, substituting mechanical user actions with automated digital processing.
2Ease of operation
If users follow standard recipe instructions without adjustment, then cooking processes are simple to follow, but cooking times and temperatures may be inaccurate when ingredient quantities or types differ from the recipe
Solution Approach 1:
The system continuously monitors the cooking environment by detecting objects and actions, compares the detected state with the predicted recipe requirements, and provides real-time feedback through augmented reality overlays. This feedback loop allows users to adjust cooking parameters based on actual ingredient quantities and cooking progress.
Solution Approach 2:
The system predicts the recipe being prepared in advance by analyzing detected ingredients and tools before cooking begins. This preliminary identification allows the system to prepare appropriate cooking instructions and adjustments for ingredient substitutions or quantity variations before the user starts cooking.
3Adaptability or versatility
If users need to search for ingredient substitutions or alternative recipes when missing ingredients, then they can continue cooking, but the process becomes complex and time-consuming
Solution Approach 1:
The system proactively identifies missing ingredients by comparing detected objects with the predicted recipe requirements and provides substitution suggestions before the user realizes they need alternatives. This preliminary detection and suggestion process eliminates the need for users to manually search for substitutions when they encounter missing ingredients.
Solution Approach 2:
The augmented reality display serves as an intermediary between the user and recipe information, presenting substitution suggestions and alternative recipes directly in the user's field of view. This intermediary interface simplifies the complexity of ingredient substitution by presenting curated options rather than requiring users to navigate through multiple information sources.
4Ease of operation
If users lack real-time guidance on proper cooking techniques and forms, then they can complete cooking tasks independently, but they cannot identify or correct improper techniques
Solution Approach 1:
The system provides real-time feedback on cooking technique by detecting user actions through pose estimation and comparing them with proper technique standards. Augmented reality overlays display corrective guidance when improper techniques are detected, such as incorrect knife handling or cooking posture, allowing users to improve their skills while cooking.
Solution Approach 2:
The patent replaces traditional cooking instruction methods (recipes, videos, or in-person guidance) with automated computer vision-based technique analysis. The system uses machine learning models to detect and evaluate cooking actions, substituting mechanical human instruction with automated visual feedback through augmented reality.
Data Source
AI summary
A client device, or an online system communicating with the device, receives video data depicting a field of view of a display area of the device and applies machine-learning algorithms to the video data to detect objects, including portions of a body of a user of the device, within the field of view and to determine a series of body poses. The device/system uses machine-learning models to predict an action performed by the user based on the series of poses and to predict a recipe being prepared based on the objects and a predicted series of actions performed by the user. The device/system selects a suggestion associated with preparing the recipe based on candidate suggestions associated with preparing the recipe, the objects, or the predicted series of actions, and generates an augmented reality element describing the suggestion. The augmented reality element is displayed in the display area of the device.