Information processing device, information processing method, and program

Abstract

The present technology relates to an information processing device, an information processing method, and a program that make it possible to appropriately present the details of cooking tasks. An information processing device according to one aspect of the present technology acquires recorded data in which the state of cooking performed by a cook is recorded, and generates, on the basis of the recorded data, data containing information relating to tasks based on difficulty level as navigation data that is used for presenting the details of a task at each step when reproducing the cooking that was recorded. The present technology can be applied to a system that assists in cooking in a kitchen.

Description

Information Processing Apparatus, Information Processing Method, and Program

[0001] The present technology particularly relates to an information processing apparatus, an information processing method, and a program that enable appropriate presentation of work content in cooking.

[0002] Recipes that ordinary cooks refer to generally contain many expressions based on tacit knowledge and ambiguous descriptions. Therefore, the cooking process varies depending on differences in cooks' skills and interpretations, differences in cooking utensils and cooking environments, etc., and as a result, the reproducibility of the finished dish decreases.

[0003] Patent Document 1 describes a cooking support system that defines the target state of ingredients in each cooking step by parameters based on sensor data that can be measured during cooking, and enables reproduction of dishes using recipe data including such parameters. During reproducible cooking, navigation that guides work in each step is presented to the cook using images, audio, and the like. By feeding back information obtained from sensor data measured during reproduction cooking to the cook, and controlling heating of ingredients until the state defined as the target state is reached, it becomes possible to reproduce the same dish as that prepared by a chef.

[0004] International Publication No. WO 2023 / 182197, Japanese Unexamined Patent Application Publication No. 2019-201339

[0005] For people who are not accustomed to cooking, it is difficult to perform work according to the navigation. For example, if the pace of adding ingredients is too fast, or the stirring speed is too fast, it becomes difficult to perform these operations while following the navigation.

[0006] The present technology has been made in view of such circumstances, and aims to enable appropriate presentation of work content in cooking.

[0007] One aspect of this technology is an information processing device which includes an acquisition unit that acquires recorded data that records the state of cooking performed by a cook, and a generation unit that generates data based on the recorded data that contains information about the tasks according to their difficulty level, to be used as navigation data to present the content of each step of the cooking process when the recorded cooking is reproduced.

[0008] In one aspect of this technology, recorded data is acquired that documents the state of cooking performed by a cook. Based on this recorded data, navigation data is generated that contains information about the tasks according to their difficulty level, and is used to present the content of each step of the cooking process when reproducing the recorded cooking.

[0009] This is a diagram showing the processing flow in a cooking support system. This is a diagram showing an example of a cooking process. This is a diagram showing an example of the configuration of a cooking support system. This is a block diagram showing an example of the functional configuration of a cooking support system. This is a flowchart showing the processing during recording cooking. This is a block diagram showing an example of the functional configuration of an information processing server. This is a block diagram showing an example of the configuration of an editing processing unit. This is a flowchart showing the editing process. This is a diagram showing an example of the recipe editor screen when recording data is imported. This is a diagram showing examples of recognition data, equipment operation data, and sensor data. This is a diagram showing an example of a divided section. This is a diagram showing an example of structured data. This is a flowchart showing the data conversion process performed in step S102 of Figure 8. This is a diagram showing an example of a process division. This is a flowchart showing the process discrimination process performed in step S103 of Figure 8. This is a diagram showing an example of the recipe editor screen after the cooking process has been determined. This is a diagram showing an example of the recipe editor screen. This is a diagram showing an example of the recipe editor screen after the cooking process has been corrected. This is a diagram showing an example of the recipe editor screen. This is a flowchart showing the level determination and data generation process performed in step S106 of Figure 8. This is a flowchart showing the level determination and data generation process when focusing on a cooking process that includes the "adding" operation. This is a flowchart explaining the level determination and data generation process when focusing on a cooking process that includes the "stirring" operation. This figure shows an example of the recipe editor screen after generating navigation data. This figure shows an example of the video editing screen. This flowchart shows the processing during the reproduction cooking process. This figure shows an example of the menu selection screen. This figure shows an example of the screen used to confirm ingredients. This figure shows an example of the cooking process confirmation screen. This figure shows an example of the cooking navigation screen. This figure shows an enlarged view of the time-series data area. This figure shows an example of the cooking navigation screen after a predetermined time has elapsed. This figure shows an example of the cooking navigation screen after a predetermined time has elapsed. This figure shows the transitions of the cooking navigation screen. This figure shows an example of the display of supplementary information. This figure shows an example of the configuration of the cooking support system. This figure shows an example of the configuration of the cooking support system. This is a block diagram showing an example of the configuration of computer hardware.

[0010] The following describes the forms for implementing this technology. The explanation will proceed in the following order: 1. Overview of the technology 2. Cooking support system 3. Recorded cooking 4. Generation of navigation data 5. Reproduced cooking 6. Variations

[0011] <<Overview of this technology>> In reproduction cooking using a cooking support system to which this technology is applied, cooking navigation is presented to the cook using a screen that includes information about the cooking process. As information about the cooking process, text and images that instruct the content of the work are presented. Information is presented to the cook not only on the screen but also using voice as appropriate.

[0012] Navigation data, which is used to present cooking navigation, is generated based on sensor data acquired during recorded cooking. Navigation data is generated after processes such as process identification, which divides the entire cooking process into multiple cooking steps, and setting target states and work contents for each cooking step.

[0013] Some tasks in recorded cooking are only possible with the expert skills of a top-class chef. Even if information on such tasks were presented as cooking instructions, it may be difficult to follow the instructions precisely. For example, if the speed of adding ingredients or stirring is too fast, it would be difficult for someone unfamiliar with cooking to follow the instructions.

[0014] Therefore, while maintaining the target state of each cooking step, it is necessary to transform the work of a top-class chef into a process that can be replicated even by someone unfamiliar with cooking.

[0015] Inconsistencies in determining the timing of cooking process breaks or variations in the wording used to instruct tasks make it difficult to provide accurate cooking navigation. Ideally, the determination of cooking process breaks and the generation of instructional wording should be automated (not dependent on human intervention).

[0016] This technology provides a recipe editor, which is computer-executable software. According to the recipe editor, time-series data such as sensor data acquired during recorded cooking is used as input information, and navigation data is generated as output information to enable even people unfamiliar with cooking to reproduce the same cooking as recorded.

[0017] The recipe editor primarily provides the following functions: • A function to determine the timing and content of each cooking step. • A function to generate instructions for the tasks involved in each cooking step. • A function to determine the timing for transitioning to the next cooking step. • A function to edit complex tasks into simpler ones. • A function to provide information about tasks tailored to the skill level of the person performing the recipe.

[0018] <<Cooking Support System>> <Cooking Process> Figure 1 shows the overall process flow, including recording and reproducing the cooking process. Details of each process will be described later.

[0019] As shown in Figure 1A, recording cooking is performed in the cooking system. During recording cooking, time-series data such as sensor data showing measurement results from various sensors is recorded. The recorded data, including sensor data, is provided to the information processing server as shown by arrow A1.

[0020] On the information processing server, navigation data is generated based on recorded data. Navigation data is in a format that can be processed by a computer.

[0021] Navigation data defines the content of each cooking step leading up to the completion of a dish, based on the following assumptions. Typically, a single dish is completed through multiple cooking steps, as shown in Figure 2.

[0022] 1. The entire cooking process consists of multiple cooking steps that are sequential in time. 2. For each cooking step, a target state and the content of the work are defined. 3. The work refers to the operation performed by the cook on cooking equipment and ingredients. 4. The target state indicates the conditions for completing one cooking step and moving on to the next. I. The conditions for moving to the next cooking step are defined as numerical indicators based on data measurable by the cooking system. II. The timing of reaching the conditions for moving to the next cooking step is represented in a way that can be uniquely identified on the time-series data detectable by the cooking system. 5. The target state is defined by one of the TTW (Time / Temperature / Weight) values. The TTW value is a parameter that defines the target state. The target parameter includes Temperature and Weight, which are sensor data measurable by the cooking system, as well as Time, which is measurable by the cooking system. I. Time: Elapsed time from the start of the cooking step [sec] II. Temperature: Temperature of the object to be heated or heating medium [°C] III. Weight: Weight of the object to be heated [g] 6. An initial state is defined for each ingredient used in the cooking process. I. II. The temperature of the ingredients at the time they are added to the cooking utensil, such as a pot [°C] II. The weight of the ingredients added to the cooking utensil, such as a pot [g]

[0023] In this way, for all cooking processes, the target state and initial state are defined using indicators that can be interpreted without subjectivity.

[0024] As shown by arrow A2 in Figure 1B, navigation data generated by the information processing server is provided to the cooking system. The cooking system then uses the navigation data to perform a reproduction of the dish. Through this reproduction, the same dish that was made during the recording process is reproduced, as shown by arrow A3.

[0025] Basically, the reproduction process follows these rules: 1. Perform the task at the start of each cooking step. 2. Start the next cooking step when the target state of the previous step has been reached.

[0026] Furthermore, the recreated cooking process is carried out with some operations, such as adjusting the heat of the cooking appliance, automated. The cook performing the recreated cooking process follows cooking navigation via a user interface for tasks such as adding ingredients and stirring.

[0027] By performing the reproduction cooking in the manner described above, variations in the cooking process caused by differences in the chef's skills and interpretations, as well as differences in cooking equipment and environment, are minimized, making it possible to consistently produce highly accurate reproductions of the original dish.

[0028] Furthermore, "cooking" refers to the finished product that is created through the cooking process. "Cooking" refers to the process of making food, or the act (work) of making food.

[0029] Hereafter, the cook who records the cooking process will be referred to as the "recording cook," and the cook who recreates the dish will be referred to as the "recreation cook." For example, someone who wants to recreate a dish prepared by a top-class chef who is the recording cook will become the recreation cook, and the dish will be recreated.

[0030] <Configuration of the Cooking Support System> Figure 3 shows an example of the configuration of the cooking support system.

[0031] The cooking support system 1 is configured by connecting a cooking system 11 and an information processing server 12 via a network such as the Internet. Various types of information are sent and received between the cooking system 11 and the information processing server 12.

[0032] The cooking system 11 is a so-called smart kitchen. In the cooking system 11, cooking is performed using cooking utensils such as pots and pans. Cooking utensils are tools used by a cook, such as pots, pans, and spatulas. Cooking equipment includes not only cooking utensils but also electrically driven equipment such as induction cookers and microwave ovens.

[0033] The cooking system 11 is configured by connecting a display 31, a speaker 32, a camera 33, a microphone 34, lighting equipment 35, a cooking environment measuring instrument 36, and a router 37 to an induction cooktop 21. Each device is connected to the induction cooktop 21 by wired communication or by wireless communication such as Wi-Fi.

[0034] The induction cooker 21 heats cooking utensils 22 such as pots and pans. Figure 3 shows a pot as an example of cooking utensil 22. A spatula, a cooking utensil used to stir the contents of the pot, is also shown.

[0035] The induction cooker 21 is equipped with a processor 51, a temperature sensor 52, a force sensor 53, an acoustic sensor 54, storage 55, and a human presence sensor 56.

[0036] The processor 51 is a computing unit that controls the heating of the cooking appliance 22. The processor 51 also controls each device that makes up the cooking system 11 and communicates with external devices and the information processing server 12. Communication with the information processing server 12 is performed via the router 37. The processor 51 executes a predetermined program and controls the overall operation of the cooking system 11.

[0037] Furthermore, the processor 51 detects instructions and actions from the cook that are input via an input UI such as the microphone 34. The processor 51 also adjusts the brightness of the lighting equipment 35 and controls the shooting conditions of the camera 33.

[0038] The temperature sensor 52 measures the temperature of the pot or the temperature of the food being cooked inside the pot. In this case, the pot acts as the heating medium, and the food being cooked inside the pot acts as the object being heated. For example, a radiation thermometer (infrared sensor) that has good responsiveness and can measure without contact may be used as the temperature sensor 52. The temperature sensor 52 may also be configured to measure both the temperature of the pot and the temperature of the food being cooked.

[0039] The force sensor 53 measures the force or weight applied to cooking utensils such as pots. For example, a piezoelectric sensor is used as the force sensor 53. The force sensor 53 is installed, for example, under the top plate of the induction cooker 21.

[0040] The acoustic sensor 54 collects sounds emitted by food and cooking utensils during cooking.

[0041] The storage 55 stores various types of data, such as programs executed by the processor 51.

[0042] The human motion sensor 56 is a sensor that uses infrared rays or the like. The human motion sensor 56 detects a cook who is near the IH cooker 21.

[0043] The display 31 displays various screens under the control of the processor 51. For example, the display 31 is provided near the IH cooker 21. Various types of information are presented to the cook using the display of the display 31. The display 31 is constituted by a display device such as a liquid crystal monitor or a projector. A wearable display such as AR glasses may be used as the display 31.

[0044] The speaker 32 outputs various sounds under the control of the processor 51. For example, an audio guide for work navigation is presented to a user using the speaker 32.

[0045] The camera 33 captures an image of the state of an object to be cooked in a pot. For example, the camera 33 is installed above the IH cooker 21 with its angle of view directed toward the pot. The moving image captured by the camera 33 is transmitted to the processor 51 in real time. As the camera 33, for example, a normal visible light camera is used. Not only a visible light camera, but also a thermography camera, a hyperspectral camera, and a depth sensing camera may be installed as the camera 33. In addition, the camera 33 captures an image of the cook performing work. The camera 33 may be constituted by a plurality of cameras, such as a camera that captures the state inside the pot and a camera that captures the cook.

[0046] As a device for inputting various types of information to the cooking system 11, for example, a smartphone (not shown) is used in addition to the microphone 34. Other mobile terminals such as a tablet terminal may also be used for input. Information input by a cook using a smartphone is acquired, for example, by the processor 51. Control of each device is appropriately performed by the processor 51 in accordance with input by the cook. A touch panel provided on the display surface of the display 31 is also used as an input device.

[0047] The cooking environment measuring instrument 36 is a measuring instrument equipped with sensor devices such as a temperature sensor, a humidity sensor, an acoustic sensor, and an atmospheric pressure sensor. The cooking environment measuring instrument 36 measures the temperature, humidity, environmental sound, and atmospheric pressure of the cooking environment. Sensor data indicating the measurement results of the cooking environment measuring instrument 36 is also supplied to the processor 51 and used for various types of processing.

[0048] As described above, the cooking system 11 is provided with various sensor devices. In the cooking system 11, for example, an image of a cook mixing food ingredients using a spatula is acquired by the camera 33.

[0049] In the cooking system 11, the position and angle of the spatula are estimated based on the image captured by the camera 33. For example, an inference model generated by machine learning using a data set of various spatula shapes is prepared in the processor 51. An inference model that receives an image captured by the camera 33 as input and outputs information on the position and angle of the spatula is prepared in the processor 51 and used for estimating the position and angle.

[0050] In the cooking system 11, the trajectory and speed of the mixing operation are acquired based on the position and angle of the spatula. Further, the start point and end point of the mixing operation, and the position in the depth direction are estimated based on the position and angle of the spatula. An AR marker or the like may be attached to the spatula, and the position and angle of the spatula may be measured by analyzing an image obtained by capturing the AR marker.

[0051] A Depth sensor may be installed on the ceiling side of the cooking environment. By using the measurement result from the Depth sensor, information in the height direction of cooking utensils such as a spatula can be acquired, and it becomes possible to estimate the three-dimensional movement of the cooking utensil based on the acquired information. By using depth information in addition to the position information of the cooking utensil, it becomes possible to estimate whether the cook is mixing the surface layer portion of the pot or the vicinity of the pot bottom.

[0052] Furthermore, by using the force sensor 53, it becomes possible to acquire the force level (intensity) of the cook's work as data.

[0053] The amount of force applied by the cook may be measured using other sensors such as piezoelectric sensors, optical sensors, strain gauges, capacitance sensors, and weight sensors. The sensors used to measure the amount of force may be installed on cooking utensils such as pots, spatulas, and chopsticks, rather than on the induction cooktop 21. Data indicating the amount of force applied by the cook, measured using force sensors, etc., is managed in synchronization with images captured by, for example, the camera 33. By synchronizing the information on the amount of force with the images captured by the camera 33, it becomes possible to improve the accuracy of the start and end points of the mixing process.

[0054] As shown in the speech bubble in Figure 3, by equipping a cooking utensil such as a spatula with an acceleration sensor, angular velocity sensor, position sensor, and attitude sensor (IMU), it becomes possible to estimate the mixing direction and speed in three dimensions.

[0055] The cooking device may be equipped with sensors for measuring the state of the food being cooked, including ingredients, such as component sensors, viscometers, and pH sensors. These sensors can be mounted, for example, using fixed or adjustable fixtures in a position that does not interfere with the addition of ingredients or stirring. This makes it possible to measure the salt concentration, sugar content, acidity, viscosity, pH, and other properties of the food being cooked.

[0056] In a cooking system 11 having such a configuration, a recording of a dish is performed. Furthermore, a reproduction of the dish is performed to recreate it. The recording and reproduction may be performed within the same cooking system, or they may be performed within different cooking systems. The following description will focus on the case where the recording and reproduction are performed using the same cooking system 11.

[0057] Figure 4 is a block diagram showing an example of the functional configuration of the cooking support system 1.

[0058] In the cooking system 11 used for recording and reproducing cooking, a recording processing unit 11A and a reproduction processing unit 11B are implemented, as shown in Figure 4. The recording processing unit 11A and the reproduction processing unit 11B are implemented by the execution of a predetermined program by the processor 51.

[0059] The recording unit 11A controls the processing during the recording cooking process. The recorded data generated by the processing during the recording cooking process is transmitted to the information processing server 12.

[0060] The reproduction processing unit 11B controls the processing during the reproduction cooking process. The processing during the reproduction cooking process is performed using navigation data generated in the information processing server 12 and provided to the cooking system 11. The navigation data provided from the information processing server 12 is stored in the storage 55 of the cooking system 11.

[0061] On the other hand, in the information processing server 12, which functions as an information processing device, an information processing unit 12A is implemented. The information processing unit 12A is implemented by the execution of a predetermined program by the processor of the computer that constitutes the information processing server 12.

[0062] The following sections will explain the recording of the cooking process, the generation of data for navigation, and the reproduction of the cooking process in order.

[0063] <<Record Cooking>> Refer to the flowchart in Figure 5 to explain the process during recording cooking.

[0064] Step S1: The recording processing unit 11A of the new recipe creation cooking system 11 creates a new file for the dish to be recorded. The recording processing unit 11A records information related to the recorded cooking, for example, based on input from the recording cook. The recording cook inputs their own information, ingredient information, etc., using the screen displayed on the display 31, for example.

[0065] Step S2: The cooking environment measurement and recording processing unit 11A measures the cooking environment and acquires information on the specifications and characteristics of cooking equipment. For example, the recording processing unit 11A acquires information on the specifications of the induction cooker 21, the pots and pans used as cooking utensils 22, and other equipment from the storage 55 or the database provided on the information processing server 12.

[0066] The storage 55 and the database of the information processing server 12 contain information on equipment such as induction cookers that the cooking system 11 supports, provided by the manufacturers and managed in conjunction with model number information. The recording processing unit 11A acquires, for example, the model number information of each piece of equipment automatically, or based on input from the recording cook to a UI such as a screen displayed on the display 31, and acquires information on the cooking equipment by referring to the database based on the acquired model number information. The information acquired by the recording processing unit 11A is recorded as information on the cooking equipment. During cooking environment measurement, information on the specifications of the induction cooker 21, the specifications of the pot, the individual characteristics of the pot, and the types / specifications of other cooking utensils is recorded.

[0067] Step S3: Start of cooking At the start of cooking, the recording processing unit 11A records information about the initial state of the cooking equipment based on information measured by various sensors. For example, information about the initial temperatures of the induction cooker 21 and the pot is recorded.

[0068] The initial temperatures of the induction cooker 21 and the pot are measured using a temperature sensor 52 installed in the cooking system 11 and acquired by the recording processing unit 11A. For example, the temperature of the cooking surface in contact with the pot is recorded as the temperature of the induction cooker 21. If measurable, the internal temperature of the induction cooker 21 is also recorded.

[0069] Step S4: During cooking, the recording processing unit 11A records various information as time-series data, such as sensor data measured by sensors installed in the cooking system 11. For example, the initial temperature of the ingredients, the weight of the ingredients added, the TTW profile, images of the inside of the pot during cooking, the position / movement of the pot during cooking, ambient temperature / humidity, and the heat setting of the stove are recorded. In addition, images of the recording cook's work, and data such as acceleration and angular velocity measured by sensors installed in cooking utensils such as spatulas are recorded.

[0070] Here, we will explain how to record the main information.

[0071] Information regarding the heat setting of the induction cooker 21 is acquired and recorded as equipment operation information. In this case, the recording processing unit 11A operates the control panel or other interface provided as the UI of the cooking system 11 to detect and acquire the heat setting value set by the recording cook.

[0072] A TTW profile, which is time-series data of time, temperature, and weight, is generated and recorded by the recording processing unit 11A. The time that constitutes the TTW profile is measured by a real-time clock built into the processor 51. As the temperature that constitutes the TTW profile, for example, the temperature of the bottom of the pot is measured by a temperature sensor 52 built into the IH cooker 21. As the weight that constitutes the TTW profile, for example, the weight of the food in the pot is measured by a force sensor 53 built into the IH cooker 21.

[0073] Images of the inside of the pot are acquired and recorded by the recording processing unit 11A. For example, a video showing the inside of the pot, captured by the camera 33, is recorded.

[0074] The initial temperature of the ingredients is obtained and recorded as information about their initial state. The initial temperature of the ingredients is the temperature of the ingredients at the time they are added to the pot (just before they are added). For example, the temperature of the ingredients is measured by the cook using a cooking thermometer.

[0075] The weight of the ingredients added is acquired and recorded as initial state information. For example, the weight and timing of the ingredients actually added to the pot are measured by the force sensor 53 and used for recording.

[0076] The above information is recorded repeatedly during the cooking process.

[0077] Step S5: The cooking completion recording processing unit 11A records the time when cooking is completed and stops recording sensor data and other information. The data, including all the information recorded in this manner, is transmitted to the information processing server 12 as recorded data and is used to generate navigation data. The recorded data is information that includes time-series data showing various states during the recorded cooking by the recording cook.

[0078] <<Generation of Navigation Data>> <Configuration of Information Processing Server> Figure 6 is a block diagram showing an example of the functional configuration of the information processing server 12.

[0079] As shown in Figure 6, the information processing unit 12A includes an editing processing unit 101, a display control unit 102, a navigation data management unit 103, and a communication control unit 104.

[0080] The editing processing unit 101 generates and saves navigation data based on the recorded cooking data generated by the recorded cooking process. The generation of navigation data by the editing processing unit 101 is performed by editing the cooking process recorded by the recorded cook so that even people unfamiliar with cooking can reproduce it, and generating data to instruct on the content of the edited cooking process. For example, as information on the tasks of a certain cooking process, navigation data is generated that includes information on tasks according to their difficulty level. Details of the editing processing unit 101 will be described later.

[0081] The display control unit 102 controls the display of the recipe editor screen. The recipe editor screen is used for generating navigation data. The generation of navigation data is managed, for example, by a navigation data provider. The generation of navigation data may be performed by the record cook or by other users.

[0082] Users who generate navigation data access the information processing server 12 by operating their user terminal and generate navigation data using the recipe editor screen displayed by the display control unit 102. Various devices such as PCs, smartphones, and tablet terminals can be used as user terminals.

[0083] The navigation data management unit 103 manages the provision of navigation data generated and stored by the editing processing unit 101. The navigation data managed by the navigation data management unit 103 is provided to the cooking system 11 (reproduction processing unit 11B).

[0084] The communication control unit 104 communicates with external devices such as the cooking system 11 and user terminals, in accordance with the control of other processing units. For example, the communication control unit 104 receives recorded data transmitted from the recording processing unit 11A. The communication control unit 104 also transmits navigation data to the reproduction processing unit 11B, in accordance with the control of the navigation data management unit 103.

[0085] Figure 7 is a block diagram showing an example configuration of the editing processing unit 101. The various components shown in Figure 7 are realized when the recipe editor is executed.

[0086] The editing processing unit 101 includes a recording data acquisition unit 121, a data conversion unit 122, a process discrimination unit 123, a process correction unit 124, a basic information correction unit 125, a navigation data generation unit 126, a navigation data correction unit 127, and a navigation data storage unit 128. The words "Load," "Process," "Tool," "Ingredient," "General Info," "Navigation," and "Save" shown below each block indicate that the processing performed by each unit is carried out when tabs T1 to T7 are operated on the recipe editor screen described later.

[0087] <Operation of the Information Processing Server> Figure 8 is a flowchart showing the overall flow of the editing process. Details of each step will be described later.

[0088] Step S101: Data acquisition The data acquisition unit 121 acquires the data generated by the recorded cooking process. The data includes pre-cooking information acquired by the cooking system 11 before the start of the recorded cooking process, and cooking-time information acquired by the cooking system 11 during the recorded cooking process.

[0089] Step S102: Data conversion processing. The data conversion unit 122 performs data conversion processing to convert the recorded data, including pre-cooking information and cooking-time information, into structured data. Structured data is data that structures information such as ingredients, cooking utensils, work, cooking time, temperature, weight, amount of water evaporation, and stirring method included in the pre-cooking information and cooking-time information. Structured data is generated by associating information from the same time. The structured data generated by the data conversion processing is temporarily stored in the data conversion unit 122.

[0090] Step S103: Process determination process The process determination unit 123 performs process determination processing and determines the timing of the cooking process divisions based on structured data. Through the process determination processing, the entire cooking process of the recorded cooking is divided into multiple cooking processes.

[0091] Step S104: The cooking process correction unit 124 corrects the cooking process identified by the process discrimination process. For example, the timing of the end of each cooking process is corrected according to manual operation by the user.

[0092] Step S105: Correction of basic information. The basic information correction unit 125 corrects the basic information contained in the structured data. For example, the basic information is corrected in response to manual operations by the user.

[0093] Step S106: Level determination and data generation process. The navigation data generation unit 126 performs level determination and data generation processing to determine the difficulty level of each cooking step. The navigation data generation unit 126 also edits the tasks of each cooking step into simpler tasks of varying difficulty levels and generates navigation data to be used for instructing the edited tasks. The difficulty level of the tasks in the cooking steps is also the level of proficiency required of the recreated chef. Hereinafter, the difficulty level of the tasks in the cooking steps will be referred to as the recreated level.

[0094] Step S107: Correction of navigation data. The navigation data correction unit 127 corrects the navigation data. The correction of the navigation data is performed, for example, in response to manual operation by the user.

[0095] Step S108: Saving Navigation Data The navigation data storage unit 128 stores navigation data. The navigation data stored in the navigation data storage unit 128 is provided to the reproduction processing unit 11B and used during reproduction cooking to present cooking navigation to the cook and to control the IH cooker 21.

[0096] The following describes the details of each step in Figure 8.

[0097] <Step S101: Importing Recorded Data> -Screen Display Example Figure 9 shows an example of the recipe editor screen when importing recorded data. The screen shown in Figure 9 will be displayed on the user terminal's display. The same applies to other screens of the recipe editor described later.

[0098] Tabs T1 through T7 are displayed at the top of the recipe editor screen. Importing recorded data is done with tab T1 selected. Below the row of tabs T1 through T7, selection buttons 201 through 203 are displayed.

[0099] The selection button 201 displayed next to the word "Video" is used to select a file containing video data recorded during the cooking process. The file containing the video data is included in the information acquired during cooking. When the selection button 201 is operated and a file is selected, the video data recorded during the cooking process is imported.

[0100] The selection button 202 displayed next to the words "Sensor log" is used to select a file containing sensor data measured during recording cooking. The file containing the sensor data is included in the information acquired during cooking. When the selection button 202 is operated and a file is selected, the sensor data measured during recording cooking is imported.

[0101] The selection button 203, displayed next to the words "Process memo," is used to select a file containing pre-cooking information. When the selection button 203 is pressed and a file is selected, the pre-cooking information is imported.

[0102] A preview area 224 is provided below the selection buttons 201 to 203. The preview area 224 displays the captured information, such as the text of the instructions entered by the cook during the recording process, audio / sound data, a graph showing the operation of the IH cooker 21, a graph showing sensor data, still images, and videos. In the example in Figure 9, a video taken during the recording process is displayed.

[0103] Examples of recorded data: Pre-cooking information includes at least one of the following: • Ingredients used • Weight of ingredients • Storage location of ingredients (room temperature, refrigerated, frozen, etc.) • Cooking utensils used • Name or ID of the chef (record cook) • Photographs (still images) of ingredients, cooking equipment, and the record cook

[0104] Information acquired during cooking includes, for example, at least one of the following: • Heat output (power input) of the induction cooker 21 • Current value of the induction cooker 21 • Load resistance value of the induction cooker 21 • Temperature of the bottom of the pot • Temperature inside the pot • Internal temperature of the ingredients • Temperature distribution • Height of the ingredients • Load • Vibration • Weight • Moisture content • Moisture evaporation rate • Cooking sounds • Ambient sounds • Statements made by the recording cook • Acceleration, velocity, and displacement during stirring • Video of the cooking process • Video of the plating process • Video of the recording cook as the subject

[0105] By performing recognition processing on videos of cooking processes, plating processes, and videos of the recording cook, information such as the ingredients used, cooking utensils, and the type of work performed by the recording cook during the cooking process is recognized. The recognition data showing these recognition results is also time-series data. The recognition processing is performed, for example, by the recording data acquisition unit 121.

[0106] The time-synchronized pre-cooking acquisition information and sensor data may be used in conjunction with the recognition process. This makes it possible to improve the accuracy of the recognition data. The recognition process may be performed by the recording processing unit 11A after the recording of cooking, and provided to the information processing server 12 along with the information acquired during cooking.

[0107] In recognizing food ingredients, broad categories such as meat, vegetables, and fish are recognized, along with intermediate and minor categories. The intermediate category of meat includes classifications such as chicken, beef, pork, and lamb. The minor category of meat includes classifications such as pork belly, minced meat, and bone-in meat.

[0108] In recognizing cooking utensils, broad categories such as spatulas and pots are recognized, along with intermediate and minor categories. The intermediate category for pots includes classifications such as woks, yukihira pots, stockpots, sauté pans, frying pans, and saucepans. The minor category for pots includes model number information. For pots, the pot body and the pot lid are recognized as separate cooking utensils.

[0109] In recognizing the actions (movements) of the recording cook, basic actions such as adding ingredients, removing ingredients, stirring, waiting, opening the lid, closing the lid, setting the pot, and removing the pot are recognized. Along with the basic actions, detailed actions are also recognized. For adding ingredients, the addition location and the position of the ingredients in the pot are recognized as detailed actions. For stirring, detailed actions such as the mixing area, mixing speed, stroke, and mixing direction are recognized.

[0110] Based on the voice data captured as information acquired during cooking, the intention of the recording cook may be recognized (estimated), and the recognized data indicating the recognition result may be used in subsequent processing. For example, statements such as "stir-fry so it doesn't burn," "make sure the eggs don't harden," and "leave some clear parts of the soup" may be recorded during the recording of cooking.

[0111] The intent behind the recording cook's actions is determined by a combination of factors, including the ingredients used, cooking utensils, and the recognition results of the recording cook's actions. For example, if silken tofu is used as an ingredient, a rubber spatula is used as a cooking utensil, and stirring is recognized as the recording cook's action, and the mixing area covers the entire area and the mixing speed is 10 cm / sec or less, then the intention can be inferred as "mixing without breaking up the ingredients."

[0112] By combining information such as the ingredients used during recording, cooking utensils, and the recognition results of the recording cook's actions with sensor data, chemical changes in ingredients can be recognized. For example, if chicken wings are used as the ingredient, and the recording cook's action is recognized as "waiting," and the bottom temperature of the pot is around 80°C, it is possible to estimate that a chemical change will occur where "collagen is broken down and gelatinized."

[0113] These recognition processes utilize methods such as leveraging machine learning models pre-trained and generated using image data, leveraging LLM and Multimodal LLM, and employing multimodal methods that combine sensor data such as temperature and force with images. Note that "Multimodal LLM" is an AI model included in "LLM (Large Language Model)". In this disclosure, LLM may be an LLM that targets only text data, or a Multimodal LLM that targets multiple different types of information such as image information and audio information. Machine learning models also include LLM.

[0114] By using machine learning models such as CNNs (Convolutional Neural Networks) to recognize (infer) the basic actions of the recorded cook, it becomes possible to recognize the actions performed at each time point shown in the video captured as information during cooking. For example, a machine learning model generated by training with pre-classified training images of basic actions such as "adding ingredients" and "stirring" is used to recognize these basic actions.

[0115] Instead of using a machine learning model that obtains recognition results for each image, a time-sensitive machine learning model, such as a 3D CNN, may be used. By using a time-sensitive machine learning model to classify basic actions, an improvement in recognition accuracy can be expected.

[0116] More detailed information about ingredients, pots, cooking utensils, etc., can also be recognized using a machine learning model generated through prior training, following a similar process to classification. By using machine learning models for object detection and segmentation, objects and their locations within an image are recognized. By tracking the location of objects, information such as mixing speed and direction can be obtained.

[0117] LLM can also be used for basic actions and object recognition. By inputting still images, videos, and information specifying the expected response format into LLM, basic actions and ingredients and cooking utensils used in the cooking process can be recognized.

[0118] By combining temperature and force information with images, it becomes possible to add information about cooking intentions such as "to prevent burning" or detailed actions such as "stirring the bottom of the pot" to tasks that are simply identified as "stirring" from images. By using various sensor data and AI technology, it becomes possible to realize a function that recognizes human actions, objects such as ingredients, cooking utensils, and pots, and cooking intentions in real time or offline.

[0119] <Step S102: Data Conversion Process> - Example of time-series data Figure 10 shows examples of recognition data, equipment operation data, and sensor data included in the information acquired during cooking. The vertical direction of the figure indicates time.

[0120] In the example shown in Figure 10, the recognition data includes time-series data for ingredients, cooking utensils, and basic actions. At the start time, the ingredients "oil," cooking utensil "ingredient container," and basic action "addition" are recognized. This recognition result indicates that at the start of the recorded cooking, the recording cook added oil from the oil container to a pot or similar. The arrows shown in the time-series data of the recognition data indicate the time intervals during which the same recognition result is obtained.

[0121] The time-series data of the heat output of the induction cooker 21 is shown as equipment operation data. The horizontal direction indicates the heat output. The time-series data of the pot temperature and the salt concentration of the contents of the pot are shown as sensor data. Various types of time-series data like these are included in the information acquired during cooking.

[0122] In the data transformation process, division points are set at timings that satisfy predetermined conditions, such as when a specific change occurs in the time-series data, and structured data is generated that associates information about the division intervals. A division interval is the interval between division points on the timeline.

[0123] Figure 11 shows an example of a division section. The horizontal dashed lines in Figure 11 indicate the timing of the division points. The numbers enclosed in small circles indicate division sections #1 to #13.

[0124] In the example in Figure 11, division points are set at each timing from time t1 to t11, and division intervals #1 to #13 are set. Basically, one division interval is set between two division points. As shown in division intervals #9 and #10, multiple division intervals may be set between two division points.

[0125] For example, segment #1, which is the interval from time t1 to time t2, is the interval in which the ingredient "oil", cooking utensil "ingredient container", and basic action "addition" were recognized. Also, segment #1 is the interval in which the heat output of the induction cooker 21 was measured as 0W, a predetermined temperature was measured for the pot temperature, and the salt concentration of the contents of the pot was measured as 0%.

[0126] Furthermore, segment #3, which is the interval from time t3 to time t4, is the interval in which the ingredients "chicken + salt", cooking utensil "bowl", and basic action "add" were recognized. In addition, a predetermined heat level was measured for the IH cooker 21, a predetermined temperature was measured for the pot temperature, and the salt concentration of the contents of the pot was measured as 0.7%.

[0127] Figure 12 shows an example of structured data.

[0128] In the example shown in Figure 12, structured data is displayed that associates the information acquired during cooking for each of the divided sections #1 to #13. The recognition data includes, in addition to the ingredients (ingredients added), cooking utensils, and basic operations explained with reference to Figures 10 and 11, information on the pot / pot lid, placement, main ingredients in the pot, ingredient positions, mixing area, mixing speed, stroke, and mixing direction. The mixing area, mixing speed, stroke, and mixing direction are detailed operation information for the basic operation "stirring," respectively.

[0129] Example flowchart Figure 13 is a flowchart showing the data conversion process performed in step S102 of Figure 8.

[0130] The structured data ultimately serves as the basis for cooking navigation, providing instructions to the recreated cook. The division points are set to detect when the recording cook's work changes, or when the state of the ingredients or the cooking process, which are implicit knowledge of the recording cook, changes. The processing shown in Figure 13 is performed by focusing on each time point in order from the start time.

[0131] In step S121, the data conversion unit 122 determines whether or not there has been a predetermined change in the equipment operation data at the time of interest. For example, if the heat output of the induction cooker 21 is adjusted or the cooking system 11 is reset, it is determined that there has been a predetermined change in the equipment operation data.

[0132] If step S121 determines that there has been a predetermined change in the cooking equipment data at the time of interest, in step S122 the data conversion unit 122 sets a division point at the time of interest.

[0133] If step S121 determines that there is no predetermined change in the cooking equipment data at the time of interest, then in step S123, the data conversion unit 122 determines whether or not there is a predetermined change in the recognition data at the time of interest. For example, if there is a change in basic operation, a change in mixing range, etc., it is determined that there has been a predetermined change in the recognition data.

[0134] If step S123 determines that there has been a predetermined change in the recognition data at the time of interest, in step S124 the data conversion unit 122 sets a division point at the time of interest.

[0135] If step S123 determines that there is no predetermined change in the recognition data at the time of interest, in step S125 the data conversion unit 122 determines whether the sensor data at the time of interest satisfies predetermined conditions. For example, if the temperature of the bottom of the pot remains constant for 30 seconds, or if the rate of temperature increase decreases, the sensor data is determined to satisfy the predetermined conditions.

[0136] If step S125 determines that the sensor data at the time of interest satisfies predetermined conditions, in step S126 the data conversion unit 122 sets a division point at the time of interest.

[0137] After setting the division points in steps S122, S124, and S126, in step S127, the data conversion unit 122 determines whether or not it has focused on the recording until the end time of cooking. Similarly, if it is determined in step S125 that the sensor data does not meet the predetermined conditions, the determination in step S127 is also performed. If it is determined in step S127 that it has not focused on the end time, the process returns to step S121, and the next time is focused on, and the above process is repeated.

[0138] On the other hand, if it is determined in step S127 that attention has been paid up to the end time, in step S128 the data conversion unit 122 generates structured data by associating information for each divided section. After that, the process returns to step S102 in Figure 8, and the subsequent processing is carried out.

[0139] The conditions for setting the division points and the order in which the conditions are met are not limited to those shown in Figure 13. Supervised learning may be performed in advance using various types of information acquired during cooking, such as recognition data, equipment operation data, and sensor data, and the corresponding structured data, and the structured data may be generated using the machine learning model generated by the learning. In this case, by inputting newly acquired information acquired during cooking into the machine learning model, structured data corresponding to the information acquired during cooking is generated.

[0140] The recognition data, equipment operation data, sensor data, and target state data, which are divided as described above, are integrated along with time information (timeline information) and stored as structured data. Because it is associated with time information, it becomes possible to play videos synchronized with each piece of information.

[0141] The recognition data that makes up structured data includes information such as ingredients, cooking utensils, and the work of the record cook. Recognition data includes at least one of the following types of information: type of pot, ingredients added, location of addition, ingredients being cooked, position of ingredients, size of ingredients, thickness of ingredients, fat content of ingredients, cooking utensils, basic movements, mixing area, mixing speed, stroke, viscoelasticity, and voice.

[0142] The equipment operation data that constitutes the structured data is information related to the operation of cooking equipment installed in the cooking system 11. The equipment operation data includes, for example, information on the heat adjustment operations and sensor reset operations performed by the recording cook, which were measured during the recording cooking process.

[0143] The sensor data that constitutes the structured data consists of information measured during recording by various sensors installed in the cooking system 11. The sensors installed in the cooking system 11 include a temperature sensor 52, a force sensor 53, an acoustic sensor 54, a camera 33, and a microphone 34. Sensors not shown, such as an acceleration sensor, an angular velocity sensor, a depth sensor, a component sensor, a viscometer, and a pH sensor, are also installed in the cooking system 11. The sensor data includes, for example, at least one of the following types of information: - Temperature of the bottom of the pot - Temperature inside the pot - Internal temperature of the ingredients - Surface temperature of the ingredients - Temperature distribution - Height of the ingredients - Load - Vibration - Weight - Moisture content - Moisture evaporation - Cooking sounds - Ambient sounds - Recorded cook's statements - Acceleration, velocity, and displacement during stirring - Salt concentration - Sugar content, acidity, viscosity, pH - Start point, end point, minimum, maximum, and median of the above profile data

[0144] Thus, the sensor data includes at least one of the following: time-series data showing changes in the state of the food ingredients, and time-series data showing changes in the state of the cooking utensils.

[0145] The target state data that constitutes the structured data includes information such as the intent of the work extracted from the audio data during recorded cooking, the intent of the work estimated from the recognition data, and chemical changes estimated by combining the recognition data and sensor data. Sensor data values ​​such as the temperature of the bottom of the pot, the temperature inside the pot, the internal temperature of the ingredients, the surface temperature of the ingredients, temperature distribution, weight, and moisture content at the end of each segment also constitute data indicating the target state.

[0146] <Step S103: Process Determination Process> - Example of the process Figure 14 shows an example of a process delimiter set by the process determination process.

[0147] In the example shown in Figure 14, the process dividers #1 to #6 are set so that the intervals between division section #1 and division section #2, division sections #3 to #7, division section #8, division sections #9 to #11, division section #12, and division section #13 each constitute one cooking process section.

[0148] Process #1 is formed by division section #1 and division section #2, and process #2 is formed by division sections #3 to #7. Process #3 is formed by division section #8, and process #4 is formed by division sections #9 to #11. Process #5 is formed by division section #12, and process #6 is formed by division section #13.

[0149] Structured data is data that has been divided into time-series data based on changes in the cook's work during recorded cooking. If stirring is performed immediately after adding ingredients, then adding ingredients and stirring are considered a single continuous process in the actual cooking scenario. Process identification processing is performed by combining information on multiple processes that are considered to be a single continuous process.

[0150] Example flowchart Figure 15 is a flowchart showing the process discrimination process performed in step S103 of Figure 8. The process in Figure 15 is performed with attention to each divided section.

[0151] In step S141, the process discrimination unit 123 determines whether the basic operation of the recognition data of the division section of interest is "input".

[0152] If the basic operation is determined to be "put in" in step S141, the process determination unit 123 sets a process delimiter at the start timing of the division section of interest in step S142.

[0153] If it is determined in step S141 that the basic operation is not "putting in", then in step S143 the process determination unit 123 determines whether the basic operation of the division section of interest satisfies the condition of being a continuous operation with the "putting in" operation of the immediately preceding division section.

[0154] If the conditions for continuous operation are met in step S143, the process determination unit 123 does not set a process delimiter in step S144.

[0155] On the other hand, if it is determined in step S143 that the conditions for continuous operation are not met, in step S145 the process determination unit 123 determines whether the basic operation of the division section of interest meets the conditions for continuous operation with the "stirring" operation of the immediately preceding division section.

[0156] If the conditions for continuous operation are met in step S145, the process determination unit 123 does not set a process delimiter in step S146.

[0157] On the other hand, if it is determined in step S145 that the conditions for continuous operation are not met, the process determination unit 123 sets a process delimiter in step S147.

[0158] After the processing in steps S142, S144, S146, and S147, in step S148, the process determination unit 123 determines whether or not all division sections have been considered. If it is determined in step S148 that not all division sections have been considered, the process returns to step S141, and the process is repeated by focusing on the next division section. If it is determined in step S148 that all division sections have been considered, the process returns to step S103 in Figure 8, and the subsequent processing is carried out.

[0159] The process shown in Figure 15 sets a process break at the start timing of the divided section of the basic operation "Input" (step S142). Process breaks #1, #2, #4, and #6 in Figure 14 are process breaks set at the start timing of the divided section of the basic operation "Input". During reproduction cooking in the cooking system 11, the addition of ingredients must be performed by the reproduction cook, so a process break is set at the start timing of the divided section of the basic operation "Input".

[0160] On the other hand, while opening and closing the pot lid and stirring are also tasks performed by the simulated cook, these tasks can be given more appropriate instructions if they are treated as a series of operations together with adding the ingredients. When focusing on a segment following the segment of the basic operation "adding ingredients," if the segment of focus satisfies the following conditions, for example, the segment of the basic operation "adding ingredients" and the segment of focus can be grouped together as the same cooking process segment (step S144). - The cooking utensil in the recognized data is "pot lid" - The basic operation in the recognized data is "open" - The basic operation in the recognized data is "close" - The cooking utensil in the recognized data is "wooden spatula" - The basic operation in the recognized data is "stirring"

[0161] Furthermore, regarding the stirring operation, unless the mixing method changes, treating it as a series of operations allows for more appropriate instructions. When focusing on a segmented section of a continuous basic operation "stirring," if the segmented section of focus satisfies, for example, the following conditions, the segmented sections of the continuous basic operation "stirring" can be grouped together as sections of the same cooking process (step S146): • There is no change in the mixing area of ​​the recognized data. • There is no change in the mixing speed of the recognized data. • There is no change in the stroke of the recognized data.

[0162] On the other hand, if the mixing area, mixing speed, stroke, etc. of the recognized data change, a change in the work instructions is required, and a process break is set (step S147). The mixing area, mixing speed, and stroke are recognized data that indicate how to mix. Process break #5 in Figure 14 is a process break set at the start timing of the division section of the basic operation "stirring" because there has been a change of a certain amount or more in the mixing area, mixing speed, and stroke of the recognized data.

[0163] Thus, in the process determination process, the timing of process divisions is determined based on the type of work performed by the recording cook, as represented by the recognition data. Furthermore, conditions are pre-set to determine whether or not a process should be considered a series of operations, and the timing of process divisions is determined by including division sections that satisfy these conditions within the same cooking process.

[0164] The conditions for setting process boundaries and the order in which these conditions are met are not limited to those shown in Figure 15. Supervised learning using structured data and corresponding process boundary information may be performed in advance, and the process discrimination process may be performed using the machine learning model generated through this learning. In this case, process boundaries are set by inputting the newly generated structured data into the machine learning model.

[0165] <Step S104: Correction of Cooking Process> -Screen Display Example Figure 16 shows an example of the recipe editor screen after the cooking process has been identified. Using the screen shown in Figure 16, the contents of each automatically separated cooking process are checked and corrected manually as needed. Checking the contents of each cooking process is performed with tab T2 selected.

[0166] The numbers 1 through 6 shown on the left side of the screen represent steps #1 through #6, as explained with reference to Figure 14. Information about the divided sections that make up each cooking step is displayed on each line.

[0167] The "Start Time" and "End Time" columns display the start and end times on the timeline, while the "Ingredient" column displays information about the ingredients being added and the ingredients being cooked. The "Tool" column displays information about the cooking equipment, and the "Action" column displays information about the basic actions. The "Additional Info." column displays information such as the addition position, ingredient position, mixing area, mixing speed, stroke, and equipment operation.

[0168] Below the row of tabs T1 to T7, selection buttons 211 and 212 are displayed. Selection button 211 is operated when selecting Auto mode. Selection button 212 is operated when selecting Manual mode. By operating selection button 212 and switching the mode from Auto mode to Manual mode, the information for each cooking process can be manually corrected.

[0169] To the right of the time for each cooking step, a checkbox labeled "Power" 221 and a checkbox labeled "Temp." 222 are displayed. Activating checkbox 221 displays the power profile for the corresponding cooking step. Activating checkbox 222 displays the bottom temperature profile for the corresponding cooking step.

[0170] A Play button 223 is displayed to the right of checkbox 222. The Play button 223 is the button that is operated to start playing the video of the corresponding cooking process.

[0171] Figure 17 shows the state with checkboxes 221 and 222 for step #2 activated and video playback started. In Figure 17, the graph 231 displayed at the bottom of the screen shows the profiles of the heat level and the bottom temperature of the pot. The video displayed in the preview area 232 is a video taken during the recording of the cooking process, corresponding to the time of step #2.

[0172] Using this screen, users can correct the information for each cooking step that has been automatically set. Users can also review the details of each cooking step through graphs and videos.

[0173] In the examples in Figures 16 and 17, information such as time, ingredients, cooking equipment, basic operation, heat level, and pot bottom temperature is shown as information for each cooking process. However, other information included in the structured data may also be displayed. For example, information such as the input position, ingredient position, mixing area, mixing speed, stroke, and equipment operation may be displayed depending on the user's operation to switch the displayed items.

[0174] <Step S105: Correction of Basic Information> -Screen Display Example Figure 18 shows an example of the recipe editor screen after the cooking process has been corrected. Using the screen shown in Figure 18, the cooking utensils used in the recorded cooking are confirmed, and the information of the cooking utensils is manually corrected as appropriate. The information of the cooking utensils as shown in Figure 18 is included in the basic information. The confirmation of cooking utensils is performed with tab T3 selected.

[0175] The information shown in Figure 18 is displayed based on recognition data and information acquired before cooking. "Pod Name" displays the type of pot, manufacturer information, and size, while "Lid Name" displays the type of pot lid, manufacturer information, and size. Pressing "Parameters" for the pot and lid displays information necessary for sensing and simulation, such as thickness, outermost diameter, bottom contact area, thermal conductivity, and emissivity. "Utensil Name" displays the cooking utensil used during cooking. In addition, in the recording cooking of the cooking system 11, registered pot sets and cooking utensils are used to ensure reproducibility.

[0176] The "Image file" is an image used to display cooking navigation and to show the preparation status of ingredients and cooking utensils before starting the recreated cooking process. Any image prepared in a database or similar can be registered. The image displayed on the preparation status check screen is the image selected when the "Prep." checkbox 241 is activated.

[0177] Figure 19 shows another example of the recipe editor screen. Using the screen shown in Figure 19, the ingredients used in the recorded cooking are confirmed, and the ingredient information is manually corrected as needed. The ingredient information shown in Figure 19 is included in the basic information. Ingredient confirmation and other operations are performed with tab T4 selected.

[0178] The information shown in Figure 19 is also displayed based on recognition data and pre-cooking information. Symbols such as ABCDE are displayed in "Symbol". The "Symbol" symbols are used to present cooking navigation and to display a screen to check the preparation status of ingredients and cooking utensils before starting the recreated cooking. Symbols are used to prevent mistakes in the order in which ingredients are added or to prevent errors.

[0179] "Ingredient Name" displays the name of the ingredient, and "Storage" indicates the storage condition of the ingredient. Information such as room temperature, refrigerated, frozen, or kept warm is displayed as "Storage" information. "Amount" displays the weight of the ingredient, and "Unit" displays the number of units. "Unit" is used to display the amount of ingredient where one set constitutes one unit, rather than by weight.

[0180] The "Image file" is an image used to display cooking navigation and to show the preparation status of ingredients and cooking utensils before starting the recreated cooking process. Any image prepared in a database or similar location can be registered.

[0181] Figure 20 shows yet another example of the recipe editor screen. The screen shown in Figure 20 is used to check the basic information of a dish, and manual corrections are made as needed. Basic information of a dish, as shown in Figure 20, is included in the base information. Checking the basic information of a dish is performed with tab T5 selected.

[0182] As shown in Figure 20, various pieces of information such as "Recipe ID," "Version," "Recipe Name," "Chef's Name," "Level," "Category," "Total Time," "Total Procedures," and "Image" are displayed as basic information about the dish. In the example in Figure 20, the basic information for a dish with "Recipe Name" set to "Sichuan Mapo Tofu" is displayed.

[0183] For example, the "Recipe Name," "Chef's Name," and "Category" information are displayed based on information obtained before cooking. The "Recipe ID" and "Version" information are automatically set by the recipe editor.

[0184] "Level" indicates the level of reproduction. In the example in Figure 20, the reproduction level of "Sichuan Mapo Tofu" is set to level 3. Reproduction levels will be explained later.

[0185] "Total Time" indicates the time required to recreate the dish. In the example in Figure 20, the time required to recreate "Sichuan Mapo Tofu" is 15 minutes.

[0186] "Total Procedures" indicates the total number of cooking steps in the entire recreated dish. The number of cooking steps identified based on the structured data represents the total number of cooking steps in the recreated dish. In the example in Figure 20, the total number of cooking steps in the recreated "Sichuan Mapo Tofu" is 17.

[0187] "Image" refers to the image of the dish used on the menu selection screen. The menu selection screen is used to select the dish (menu) to be recreated during the recreated cooking process. For example, images captured from videos taken during the recording of the cooking process or from videos taken during the plating process are registered as "Image". Alternatively, the images registered as "Image" may be obtained from information acquired before cooking.

[0188] The area 251 on the right side of the screen is the preview area for the menu selection screen. Information including images registered under "Image" is displayed on the menu selection screen. Details of the menu selection screen will be described later.

[0189] The level of reproduction for each dish is defined using criteria such as the following: Level 4 (Lv.4): A person who has performed Level 3 reproduction cooking of the same menu multiple times and has not received any feedback from the cooking system 11 regarding points to note when adding ingredients or stirring can perform the reproduction cooking. Level 3 (Lv.3): A person who has passed the certification test can perform the reproduction cooking after receiving only preliminary guidance. Level 2 (Lv.2): A person who has worked in reproduction cooking using the cooking system 11 for more than one week can perform the reproduction cooking on their first try. Level 1 (Lv.1): A beginner can perform the reproduction cooking without failure.

[0190] In this way, each dish is assigned a level that indicates the overall difficulty of recreating it. Level 4 is the most difficult level to recreate, and Level 1 is the easiest level to recreate.

[0191] The certification test is a test used by people who will be using the cooking system 11 to prove their skills. It assesses whether they can add ingredients at the specified timing, and whether they can perform the stirring operation in the specified pattern, range, and speed, by actually performing the work using the cooking system 11.

[0192] <Step S106: Level Determination and Data Generation Process> - Example of basic processing Figure 21 is a flowchart showing the level determination and data generation process performed in step S106 of Figure 8. The process in Figure 21 is performed with attention to each cooking step.

[0193] In step S161, the navigation data generation unit 126 determines whether the cooking process of interest is a cooking process that includes tasks subject to level determination. In this example, a reproduction level is set for each task in the cooking process. If a task of level 4 is included, or if a task of level 3 is included, it is determined to be a cooking process that includes tasks subject to level determination.

[0194] If step S161 determines that the cooking process includes an operation that is subject to level determination, in step S162 the navigation data generation unit 126 determines whether or not the operation included in the cooking process of interest is a level 4 operation.

[0195] If it is determined in step S162 that the operation is at level 4, in step S163 the navigation data generation unit 126 generates data corresponding to the reproduction levels of level 4, level 3, and level 2 as navigation data for the cooking process of interest.

[0196] On the other hand, if it is determined in step S162 that the operation is not a Level 4 operation, then in step S164, the navigation data generation unit 126 determines whether or not the operation included in the cooking process of interest is a Level 3 operation.

[0197] If it is determined in step S164 that the operation is Level 3, in step S165 the navigation data generation unit 126 generates data corresponding to the reproduction levels of Level 3 and Level 2, respectively, as navigation data for the cooking process of interest.

[0198] If it is determined in step S164 that the operation is not a Level 3 operation, in step S166 the navigation data generation unit 126 generates data corresponding to the Level 2 reproduction level as navigation data for the cooking process of interest. Similarly, if it is determined in step S161 that the cooking process does not include the operation subject to level determination, data corresponding to the Level 2 reproduction level is generated in step S166.

[0199] After processing in steps S163, S165, and S166, in step S167, the navigation data generation unit 126 generates level 1 navigation data based on level 2 navigation data. Level 1 navigation data is generated by customizing the level 2 navigation data. For example, level 1 navigation data is generated by customizing the content of the level 2 navigation data to include information on simpler tasks and to have a larger amount of information.

[0200] After generating navigation data containing data corresponding to each reproduction level, in step S168, the navigation data generation unit 126 determines whether or not all cooking processes have been considered. If it is determined in step S165 that all cooking processes have not been considered, the process returns to step S161, and the process is repeated with attention to the next cooking process. If it is determined in step S165 that all cooking processes have been considered, the process returns to step S106 in Figure 8, and the subsequent processes are carried out.

[0201] ・Example 1 of a specific process: Referring to the flowchart in Figure 22, we will explain the level determination and data generation process when focusing on the cooking process, including the "addition" operation.

[0202] For people with little cooking experience, adding ingredients in quick succession can be a difficult task. Figure 22 shows the process when focusing on a cooking process that involves adding ingredients in succession, that is, a cooking process that involves the basic action of "adding" ingredients in succession.

[0203] In the process shown in Figure 22, if the interval between "input" operations is less than 5 seconds, it is determined to be a Level 4 operation; if it is 5 seconds or more but less than 15 seconds, it is determined to be a Level 3 operation; and if the interval between "input" operations is 15 seconds or more, it is determined to be a Level 2 operation.

[0204] In step S181, the navigation data generation unit 126 determines whether the interval between "input" operations is less than 5 seconds.

[0205] If the interval between "addition" operations is, for example, 4 seconds, it is determined in step S181 to be a cooking process that includes a Level 4 operation. In this case, in step S182, the navigation data generation unit 126 generates Level 4 data as navigation data for the cooking process of interest. The navigation data generation unit 126 also generates Level 3 data in step S183 and Level 2 data in step S184. The processes in steps S182, S183, and S184 correspond to the process in step S163 in Figure 21.

[0206] Thus, if the "adding" step included in the cooking process of interest is a Level 4 step, in addition to Level 4 navigation data, Level 3 and Level 2 navigation data are generated, which represent the "adding" step as a simpler task.

[0207] The cooking navigation presented by the Level 4 navigation data (step S182) divides each step at the same time intervals as during the recorded cooking process, and promptly instructs the user to transition to the next cooking step. The time of the cooking step of interest is used directly for the reproduction cooking process. During the reproduction cooking process, the user is instructed to transition to the next cooking step only after a predetermined amount of time has elapsed since the start of the previous cooking step.

[0208] The cooking navigation provided by the Level 3 navigation data (step S183) will give different instructions depending on whether or not there are temperature changes during the cooking process.

[0209] If there is no temperature change, the cooking navigation will instruct the user to extend the time for the "adding" step to a predetermined time, such as 15 seconds. During the reproduction cooking process, the user will be instructed to transition to the next cooking step only after a predetermined amount of time has elapsed since the start of the previous cooking step.

[0210] When there are temperature changes, the cooking navigation will reduce the heat output of the IH cooker 21 and instruct the system to transition to the next cooking step when the total amount of heat supplied to the ingredients becomes equal to the cumulative amount during the recorded cooking. The time delay in adding the ingredients will also be taken into consideration. The time delay in adding the ingredients will be detected by a sensor built into the IH cooker 21. The system will instruct the system to transition to the next cooking step when the total amount of heat supplied to the ingredients reaches a threshold amount. This ensures that the ingredients are supplied with the same amount of heat as during the recorded cooking, and makes it possible to bring the state of the ingredients at the end of the cooking process closer to that of the recorded cooking.

[0211] The cooking navigation provided by the Level 2 navigation data (step S184) will also provide different instructions depending on whether or not there are temperature changes during the cooking process.

[0212] When there is no temperature change, the cooking navigation system will instruct the next ingredient to be added when it detects whether or not there has been a change in weight due to the addition of ingredients, or when the simulated chef presses a button after adding ingredients. In the former case, the transition to the next cooking step is instructed based on the condition that a change in weight has occurred, and in the latter case, the transition to the next cooking step is instructed based on the condition that the button has been pressed. This makes it possible to transition to the next cooking step only after the addition of ingredients has been completely completed.

[0213] When there is a temperature change, the cooking navigation will lower the heat output of the induction cooker 21 and instruct the system to transition to the next cooking step when the total amount of heat supplied to the ingredients becomes equal to the cumulative amount during the recorded cooking. The transition to the next cooking step is instructed when the total amount of heat supplied to the ingredients reaches a threshold amount.

[0214] If step S181 determines that the interval between "input" operations is not less than 5 seconds, then in step S185, the navigation data generation unit 126 determines whether the interval between "input" operations is less than 15 seconds.

[0215] If the interval between "addition" operations is, for example, 10 seconds, it is determined in step S185 to be a cooking process that includes a Level 3 operation. In this case, in step S186, the navigation data generation unit 126 generates Level 3 data as navigation data for the cooking process of interest. The navigation data generation unit 126 also generates Level 2 data in step S187. The processes in steps S186 and S187 correspond to the process in step S165 in Figure 21.

[0216] Thus, if the "adding" step included in the cooking process of interest is a Level 3 step, in addition to Level 3 navigation data, Level 2 navigation data is generated, which represents the "adding" step as a simpler step.

[0217] The Level 3 navigation data (step S186) is the same as the navigation data generated by the processing in step S183. The Level 2 navigation data (step S187) is the same as the navigation data generated by the processing in step S184.

[0218] If step S185 determines that the interval between "addition" operations is not less than 15 seconds, then in step S188, the navigation data generation unit 126 generates level 2 data as navigation data for the cooking process of interest. The process in step S188 corresponds to the process in step S166 in Figure 21.

[0219] The cooking navigation presented by the Level 2 navigation data (step S188) divides each step into the same time intervals as during the recorded cooking process, and instructs the system to transition to the next cooking step when a predetermined time has elapsed. The time of the cooking step of interest is used directly for the reproduction cooking process. The system instructs the system to transition to the next cooking step only after a predetermined time has elapsed from the start of the current cooking step.

[0220] The process shown in Figure 22 generates navigation data containing information that represents the same "addition" operation as different operations depending on the level of reproduction. For cooking processes that include the Level 4 "addition" operation, information is generated for Level 4, which instructs the system to transition to the next cooking process in the same amount of time as during the recorded cooking process. Additionally, information for Level 3, which instructs the system to transition to the next cooking process in a longer amount of time than during the recorded cooking process, and information for Level 2, which instructs the system to add ingredients and press a button to transition to the next cooking process. These are all included in the navigation data. The navigation data contains multiple types of information for the same cooking process, each with a different level of reproduction.

[0221] Because the "adding" process can be performed over a longer period than the time recorded for cooking, the information for Level 3 represents the same "adding" process as a simpler task than the information for Level 4. Also, because you only need to press a button after performing the "adding" process, the information for Level 2 represents the same "adding" process as a simpler task than the information for Level 3.

[0222] During the reproduction cooking process, a reproduction cook with a reproduction level of 4 is instructed to perform the "addition" step at the same time as during the recorded cooking, based on the information for level 4, and then proceed to the next cooking step. A reproduction cook with a reproduction level of 3 is instructed to perform the "addition" step at a longer time than during the recorded cooking, based on the information for level 3, and then proceed to the next cooking step. A reproduction cook with a reproduction level of 2 is instructed to perform the "addition" step, then press the button, and then proceed to the next cooking step, based on the information for level 2.

[0223] This makes it possible to present the "addition" process as information appropriate to the level of the person recreating the dish.

[0224] ・Example 2 of specific processing: Referring to the flowchart in Figure 23, we will explain the level determination and data generation process when focusing on a cooking process that includes the "stirring" operation.

[0225] For people with little cooking experience, even stirring ingredients with a certain amount of force can be a difficult task. Figure 23 shows the process when focusing on a cooking process that includes the basic action of "stirring" with a force of a certain value or more, such as 15N.

[0226] In the process shown in Figure 23, if the mixing speed is 15 cm / sec or more and the bottom temperature of the pot is 150°C or more, it is judged as a Level 4 operation. If the mixing speed is 15 cm / sec or more and the bottom temperature of the pot is less than 150°C, it is judged as a Level 3 operation. If the mixing speed is 15 cm / sec or more, it is judged as a Level 2 operation.

[0227] In step S201, the navigation data generation unit 126 determines whether the mixing speed is 15 cm / sec or more.

[0228] If it is determined in step S201 that the mixing speed is 15 cm / sec or more, then in step S202, the navigation data generation unit 126 determines whether or not the bottom temperature of the pot is 150°C or higher.

[0229] If the bottom temperature of the pot is determined to be 150°C or higher in step S202, the navigation data generation unit 126 generates level 4 data as navigation data for the cooking process of interest in step S203. The navigation data generation unit 126 also generates level 3 data in step S204 and level 2 data in step S205. The processes in steps S203, S204, and S205 correspond to the process in step S163 in Figure 21.

[0230] Thus, if the "stirring" step included in the cooking process of interest is a Level 4 step, in addition to Level 4 navigation data, Level 3 and Level 2 navigation data are generated, representing the "stirring" step as a simpler operation.

[0231] The Level 4 navigation data (step S203) will indicate the target state using both text and audio. The target state may be presented using either text or audio, or both. Guidance on mixing speed is provided using rhythmic sounds.

[0232] The Level 3 navigation data (step S204) will show the stirring pattern corresponding to the target state using a video or other means. The number of stirring cycles will also be specified.

[0233] The Level 2 navigation data (step S205) will show a stirring pattern corresponding to the target state using a video or other means. It will also instruct the system to reduce the heating and transition to the next cooking step when the total amount of heat supplied to the ingredients becomes equal to the cumulative amount during the recorded cooking. To improve reproducibility, the stirring speed may be instructed using rhythmic sounds, regardless of the reproducibility level.

[0234] If it is determined in step S202 that the temperature of the bottom of the pot is not 150°C or higher, in step S206 the navigation data generation unit 126 generates level 3 data as navigation data for the cooking process of interest. In addition, the navigation data generation unit 126 generates level 2 data in step S207. The processes in steps S206 and S207 correspond to the process in step S165 in Figure 21.

[0235] Thus, if the "stirring" step included in the cooking process of interest is a Level 3 step, in addition to Level 3 navigation data, Level 2 navigation data is generated, which represents the "stirring" step as a simpler operation.

[0236] The Level 3 navigation data (step S206) is the same as the navigation data generated by the process in step S204. The Level 2 navigation data (step S207) is the same as the navigation data generated by the process in step S205.

[0237] If it is determined in step S201 that the mixing speed is not 15 cm / sec or higher, in step S208 the navigation data generation unit 126 generates level 2 data. The level 2 navigation data (step S208) is the same as the navigation data generated by the processing in step S206.

[0238] The process shown in Figure 23 generates navigation data containing information that represents the same "stirring" operation as different operations depending on the level of reproduction required.

[0239] Furthermore, in Level 4, the target state is presented using text and audio, while in Level 3, the target state is presented using video. In other words, the target state is presented using the type of information appropriate to the level of reproduction. Information other than the target state may also be presented to indicate the content of the "stirring" operation.

[0240] For cooking processes that include the Level 4 "stirring" operation, navigation data is generated that includes Level 4 information that presents the target state using text and audio, as well as Level 3 and Level 2 information that presents the target state using video. Furthermore, the Level 2 information instructs the user to reduce the heating and perform the "stirring" operation for a longer period than the time recorded during cooking.

[0241] As demonstrated in the video, the cook only needs to perform the "stirring" step. Therefore, the information for Level 3 and Level 2 presents the same "stirring" step as a simpler task than the information for Level 4. Furthermore, since the "stirring" step can be performed for a longer period than during the recorded cooking time, the information for Level 2 presents the same "stirring" step as a simpler task than the information for Level 3.

[0242] During the reproduction cooking process, a cook with a reproduction level of 4 will be instructed on the target state and other details using text and audio based on information for level 4. A cook with a reproduction level of 3 will be presented with the target state and other details using video based on information for level 3. A cook with a reproduction level of 2 will be controlled to reduce the heating rate of the induction cooker 21 based on information for level 2, and will also be presented with the target state and other details using video.

[0243] This makes it possible to present the "stirring" step as information appropriate to the level of the person recreating the dish.

[0244] The level of task reproduction may be determined using a predetermined threshold, or it may be determined during the reproduction cooking process. For example, based on log data of the reproduced cook's work acquired by the IH cooker 21 and surrounding sensors, tasks that are difficult for the reproduced cook may be detected, and their level may be determined. Tasks that cannot be performed as instructed, such as those with many delays, may be determined to be high-level tasks.

[0245] By evaluating the skills of the recreated chef and their proficiency with the cooking system 11, it is possible to provide the recreated chef with navigation for the optimal procedure for recreating the dish, thereby ensuring a high degree of accuracy in the reproduction.

[0246] When presenting cooking navigation by describing a highly reproducible task as a low-reproducibility task, the time required for that task usually increases, and the total time required for recreating the dish also increases. Furthermore, more cooking utensils need to be prepared. From the perspective of the time required for recreating the dish, it is preferable to present cooking navigation by describing the task as a highly reproducible task whenever possible.

[0247] <Step S107: Correction of Navigation Data> -Screen Display Example Figure 24 shows an example of the recipe editor screen after the navigation data has been generated. Using the screen shown in Figure 24, the contents of the cooking navigation for each cooking step are checked and corrected manually as needed. Checking the contents of the cooking navigation for each cooking step is done with tab T6 selected.

[0248] Below the row of tabs T1 to T7, level indicators 261-1 to 261-4 are displayed, containing the text "Level 1" to "Level 4". Level indicators 261-1 to 261-4 display the reproduction level of the tasks included in the selected cooking process. In the example in Figure 24, the most difficult task is a Level 3 task, so level indicator 261-4 is inactive.

[0249] The numbers in the "Mark" column on the left side of the screen indicate the cooking steps. Information regarding cooking navigation for each cooking step is displayed in each row.

[0250] As shown in Figure 24, information for each item such as "Trans. Condition," "Graph," "Delay," "Navigation," "Warning," "Video," and "Voice" is displayed. "Graph," "Navigation," "Warning," "Video," and "Voice" are the information presented as cooking navigation.

[0251] "Trans. Condition" indicates the conditions for transitioning to the next cooking step. This "Trans. Condition" information is not presented to the recreater.

[0252] "Graph" displays a graph of time-series data, such as sensor data.

[0253] "Delay" indicates the time set when providing continuous instructions without separating the cooking process. By setting the "Delay" time, it becomes possible to set supplementary information such as "temporarily stop stirring" or "prepare the next ingredient."

[0254] "Navigation" refers to information about instruction phrases. Instruction phrases are words that indicate the content of the instructions. "Navigation" information is generated based on recognition data, equipment operation data, sensor data, and target state data included in the structured data. For example, supervised learning is performed in advance using recognition data, equipment operation data, sensor data, target state data included in the structured data and their corresponding phrases, and a machine learning model such as an LLM generated by the learning is prepared in the navigation data generation unit 126. Instruction phrases are generated by inputting the recognition data, equipment operation data, sensor data, and target state data included in the structured data into the machine learning model.

[0255] If only recognition data is used, it is not possible to determine whether ingredients were added before or after heating. By using equipment operation data, sensor data, and target state data, it becomes possible to generate instruction text that includes the state before and after heating, as well as the target state.

[0256] "Warning" indicates a warning message.

[0257] To the right of "Warning," icons 265 for "Video" and 266 for "Voice" are displayed. Icon 265 is used to preview the video presented as cooking navigation. Icon 266 is used to preview the audio presented as cooking navigation.

[0258] On the right side of the screen, you will see buttons 262 for "Create Voice," 263 for "Custom Video," and 264 for "Custom Voice."

[0259] Button 262 is operated when generating an audio guide to be presented as cooking navigation using a cloud-based speech synthesis system. By operating button 262 and entering instruction text, the audio guide data is generated.

[0260] The "Custom Video" button 263 is operated when editing a video presented as cooking navigation. When button 263 is operated, for example, an editing screen used for editing the video is displayed.

[0261] Figure 25 shows an example of a video editing screen.

[0262] The editing screen displays selection buttons 271-1 to 271-5, which are used to select the video layout in the navigation screen. Below the selection buttons 271-1 to 271-5 is a preview area 272. The preview area 272 shows the screen displaying the video with the layout selected using the selection buttons 271-1 to 271-5.

[0263] In the example shown in Figure 25, the selection buttons 271-3 are operated, and the two-screen layout is displayed in the preview area 272. When the two-screen layout is selected, the video recorded during the cooking process is displayed on the left, and still images supplementing the instructions are displayed on the right. In this way, by displaying information that is not available in the video alone, or by displaying a mixing guide, it becomes possible to more easily convey the status of the work to the person recreating the cooking process.

[0264] Returning to the explanation of Figure 24, the "Custom Voice" button 264 is operated when editing the voice guide presented as cooking navigation. Similarly, when button 264 is operated, the editing screen used to edit the voice guide is displayed.

[0265] By using this screen, users can manually make adjustments by changing the "Trans. Condition" and "Graph" options, or by changing the text in "Navigation" and "Warning." It is also possible to set supplementary information. An example of how supplementary information will be displayed will be described later.

[0266] <Step S108: Saving Navigation Data> After the navigation data has been corrected, the navigation data storage unit 128 saves navigation data in a format that can be played back in the cooking system 11, depending on whether tab T7 (Figure 25) has been operated.

[0267] Navigation data may include at least one of the following: • Program files that control the presentation of instructions and equipment operation information • Audio data of instructions • Data for graph display based on sensor data during recorded cooking • Still images of the finished dish • Still images of ingredients and cooking utensils • Videos taken during recorded cooking

[0268] The editing process described above is performed based on the record data of each dish, and navigation data for various dishes is stored in the navigation data storage unit 128. A set of navigation data, including navigation data for various dishes, is provided to the reproduction processing unit 11B at a predetermined timing and used for reproduction cooking.

[0269] <<Recreation Cooking>> <Example Flowchart> Refer to the flowchart in Figure 26 to explain the process during recreation cooking.

[0270] The process shown in Figure 26 begins, for example, when the chef selects a dish to be recreated. The dish is selected using the menu selection screen displayed on the display 31. A specific example of the screen display will be described later.

[0271] Step S301: The reproduction processing unit 11B (Figure 4) of the cooking system 11 reads the navigation data for the selected dish.

[0272] Step S302: The cooking environment measurement and reproduction processing unit 11B measures the cooking environment and acquires the specifications and characteristics of the cooking equipment used in the reproduced cooking. The measurement of the cooking environment during the reproduced cooking is performed in the same manner as the cooking environment measurement in the recorded cooking (step S2 in Figure 5).

[0273] If the reproduction processing unit 11B detects a difference between the cooking environment of the reproduced cooking and the cooking environment of the recorded cooking, as represented by the information included in the navigation data, it presents information indicating this difference to the cook reproducing the cooking and issues a warning. The cook reproducing the cooking then changes cooking equipment or makes other adjustments to bring the cooking environment of the reproduced cooking closer to that of the recorded cooking, using the display 31 or the like to indicate the difference in the cooking environment.

[0274] Step S303: The cooking start reproduction processing unit 11B measures the temperature of the IH cooker 21 and the pot as the initial state of the cooking equipment, similar to the start of cooking in the recorded cooking (step S3 in Figure 5). If the measured temperature differs significantly from the initial state temperature recorded in the navigation data, the reproduction processing unit 11B suspends the start of heating and warns the reproduced cook.

[0275] Step S304: The cooking reproduction processing unit 11B assists the reproduction cook in cooking based on navigation data. Specifically, the reproduction processing unit 11B presents cooking navigation for each cooking step. The reproduction processing unit 11B also measures the state of the reproduced cooking in real time and controls the cooking equipment to reproduce the cooking process (time series of the TTW profile).

[0276] Step S305: Completion of Cooking. Once all cooking steps are finished, the reproduction cooking is complete, and the dish is ready. Through this series of processes, the reproduction cook can consistently produce dishes with a high degree of accuracy in reproduction. In addition, information instructing the user on the content of each cooking step is displayed on the cooking navigation screen, allowing the reproduction cook to easily proceed with the reproduction cooking.

[0277] <Example of screen display> This section explains specific examples of screens displayed on display 31 at each stage of the recreated cooking process.

[0278] • The menu selection screen, Figure 27, is an example of a menu selection screen.

[0279] The menu selection screen in Figure 27 shows the screen when "Sichuan Mapo Tofu" is selected. For example, the screen in Figure 27 is displayed based on the information set using the screen described with reference to Figure 20. The name of the dish is displayed prominently in the approximate center of the menu selection screen, and below it are the name of the chef who recreated the dish, the time it took to recreate the dish, the level of recreation, and the number of cooking steps. A large image of the dish is displayed to the left of the dish name and other information.

[0280] At the bottom of the menu selection screen, genre buttons 301 are displayed, which are used to select the type of cuisine. In the example shown in Figure 27, the buttons used to select each of the following genres—Japanese, Chinese, French, Italian, and sweets—are displayed as genre buttons 301.

[0281] Normally, dishes made with the same ingredients and procedures are treated as the same dish. However, in the cooking system 11, information specific to the recording cook, such as heating profiles and stirring methods, is also recorded. Therefore, even dishes made with the same ingredients and procedures are managed separately as different dishes depending on the recording cook. Even when the same recording cook performs the recording, their skills and preferences may differ depending on the time period. Therefore, dishes may be distinguished by the time the recording was performed, and the time period information may be clearly indicated. In addition, the recording cook's information may include their affiliation at the time of recording and their affiliation at the time of re-cooking (the present).

[0282] If the skills of the recreated chef and their proficiency with the system can be evaluated in advance, the recreation level will be displayed, allowing the chef to select a dish that matches their skill level. This ensures a high degree of accuracy in recreating the dish.

[0283] The Favorites button 302 is displayed in the lower left corner of the menu selection screen. The Favorites button 302 is used to display dishes that have been registered as favorites.

[0284] The menu selection screen displays a back button 303L and a forward button 303R at the left and right ends, respectively. The back button 303L and forward button 303R are used to switch between dishes. When the back button 303L or forward button 303R is pressed, the information of the dish, including the dish image, is changed, and information of a different dish is displayed.

[0285] The start button 304 and ingredient information button 305 are displayed near the center of the menu selection screen. The start button 304 is operated when you have finished selecting a dish and want to start the recreated cooking process. The ingredient information button 305 is operated when you want to check the ingredients to be used in the recreated cooking process.

[0286] Figure 28 shows a screen used for checking ingredients, etc. When the ingredient information button 305 is pressed, information such as the ingredients used in the recreated dish, the amount of ingredients used, the cooking process in which the ingredients are used, the storage location of the ingredients, the cooking equipment to be used, and the cooking utensils to be used is displayed.

[0287] Information about ingredients is displayed based on the settings made using the screen described with reference to Figure 19. Similarly, information about cooking utensils is displayed based on the settings made using the screen described with reference to Figure 18.

[0288] The recreater can confirm what ingredients are needed and in what quantities. They can also confirm the correct preparation state for the ingredients. Sorting displays, such as rearranging ingredients in the order of the cooking process or organizing them by storage location, may also be available. This allows the recreater to more reliably confirm the preparation status of the ingredients.

[0289] In the upper right corner of the menu selection screen in Figure 27, a language switching button 306 and a search button 307 are displayed, and below them, a favorites button 308 is displayed. The language switching button 306 is used to switch the language of the navigation. The search button 307 is used when searching for dishes by entering text. The favorites button 308 is used when registering the currently displayed dish as a favorite dish.

[0290] The menu selection screen may also be configured to allow user registration. User registration allows for the management of attribute information linked to the cook, such as the number of times they have recreated dishes in the past, the history of dishes they have made, their training status, tasks they struggle with, and their proficiency in operating equipment. This attribute information can be used, for example, to display only dishes at a level that matches the cook's skill on the menu selection screen, to automatically skip training during cooking process confirmation, or to highlight supplementary information about tasks the cook struggles with.

[0291] • Cooking process confirmation screen Figure 29 shows an example of the cooking process confirmation screen. The cooking process confirmation screen is displayed when the start button 311 in Figure 28 is pressed.

[0292] The cooking process confirmation screen is used to confirm the cooking process for reproducing the dish. Text, images, and other information describing the outline of each cooking step are presented sequentially, one step at a time. Audio from speaker 32 is also used as appropriate to describe the outline of each cooking step.

[0293] At the top of the cooking process confirmation screen, the numbers of the cooking steps are displayed in a row. Among the displayed numbers, the number of the cooking step currently being introduced is highlighted by inverting its color or other means. In the example in Figure 29, the cooking step being introduced is step 7 out of a total of 17 steps.

[0294] A rectangular display area 321 is formed below the sequence of cooking step numbers. The display area 321 displays text and other information that outlines the cooking process. In the example in Figure 29, text and a video are displayed indicating that step 7 is the process of adding doubanjiang, garlic, and douchi to the pot. For example, a video filmed during the recording of the cooking process is displayed.

[0295] The display area 321 is flanked by a back button 322L and a forward button 322R. The back button 322L is used to return to the previous cooking step. The forward button 322R is used to advance to the next cooking step.

[0296] At the top of the cooking navigation screen, information such as the name of the dish, the name of the cook who recorded it, cooking time, reproduction level, and number of steps is also displayed. In the upper right corner of the cooking navigation screen, a start button 311 is displayed, which is operated when starting the reproduction cooking process.

[0297] • Diagram 30 of the cooking navigation screen configuration is an example of a cooking navigation screen.

[0298] As shown in Figure 30, the cooking navigation screen is mainly composed of three strip-shaped areas of approximately the same height: a time-series data area 331, and cooking process areas 332-1 and 332-2. The time-series data area 331 is at the center, with the cooking process area 332-1 positioned above the time-series data area 331 and the cooking process area 332-2 positioned below the time-series data area 331. The time-series data area 331 is positioned between the cooking process areas 332-1 and 332-2. A narrow process number area 333 is formed below the cooking process area 332-2.

[0299] The display of the cooking navigation screen, which has this configuration, changes over time. The horizontal direction of the cooking navigation screen indicates the direction of time. The position indicated by the thick reference line L1 is the current time. The part to the left of the reference line L1 is displayed in gray. The fact that it is displayed in gray indicates that the information to the left of the reference line L1 is past information. The information to the right of the reference line L1 is information from the future.

[0300] In the example shown in Figure 30, section T2, indicated by the arrow, is the current cooking step, step 4, and section T3 is the next cooking step, step 5. Section T1 is the cooking step immediately preceding the current one, step 3. Only a portion of the sections for steps 3 and 5 are shown, not the entire section.

[0301] The time-series data area 331 is a display area for information indicating time-series data. Based on the graph display data included in the navigation data, a graph showing the time-series data of sensor data recorded during the recording cooking process and a graph showing the time-series data of sensor data detected during the reproduction cooking process are displayed in the time-series data area 331. Depending on the cooking process, information other than information indicating time-series data may be displayed in the time-series data area 331 as appropriate.

[0302] Figure 31 is a magnified view of the time-series data area 331.

[0303] In the example shown in Figure 31, the start time of section T2 is displayed slightly to the left of the current time. The state shown in Figure 31 is the state immediately after the start of the current cooking process (1 second later).

[0304] Graph G1, displayed in section T1, shows the time-series data of the previous cooking process, and graph G2, displayed in section T2, shows the time-series data of the current cooking process. In the example in Figure 31, the words "Mixing strength" are displayed next to the reference line L1, indicating that graph G2 is the time-series data of the mixing strength during the stirring operation. The section T2 portion of the time-series data area 331 is displayed using a conspicuous color such as orange to distinguish it from the section of the next cooking process. Graph G3, displayed in section T3, shows the time-series data of the next cooking process. Graphs G1, G2, and G3 are information displayed based on the time-series data of sensor data detected during recording cooking and recorded in the navigation data.

[0305] Graphs G1, G2, and G3, like the information regarding the next cooking step, are displayed as they move to the left over time. For example, the entirety of graphs G1, G2, and G3 moves so that the measurement results of the sensor data at the time of the recorded cooking, corresponding to the current time, are displayed at the position of the current time indicated by the reference line L1. In the state shown in Figure 31, the value of graph G2 at the current time position represents the value measured at the time of the recorded cooking as the "mixing strength" of the cooking operator's stirring work, one second after the start of the cooking step 4. In this way, the entirety of graphs G1, G2, and G3 moves to the left, switching the range of time-series data displayed based on the navigation data.

[0306] Within the time-series data area 331, graph G11 is displayed in a prominent color overlaid on other graphs in the portion to the left of the reference line L1. Graph G11 shows time-series data of sensor data detected during the reproduction cooking process. Using the current time position indicated by the reference line L1 as a reference, graph G11 is updated and displayed in real time to show the latest sensor data status at that reference position.

[0307] The portion of graph G11 displayed in section T1 shows time-series data of the same type as the sensor data shown in the time-series data of the previous cooking process (graph G1). Similarly, the portion of graph G11 displayed in section T2 shows time-series data of the same type as the sensor data of "mixing strength" shown in the time-series data of the current cooking process (graph G2).

[0308] The recreater will work to bring the current value of graph G11 closer to the value of the graph displayed based on the navigation data, while watching the real-time updated display of graph G11. In the state shown in Figure 31, the closer graph G11 is to graph G2, the better the recreater's stirring strength is replicating the stirring strength of the record cook during the original cooking process.

[0309] Returning to the explanation of Figure 30, cooking process area 332-1 and cooking process area 332-2 are display areas for information related to the cooking process. This information includes details of the work performed.

[0310] In the example shown in Figure 30, the current cooking step, Step 4, is the task of adding and stirring the beef. Instructions for adding and stirring the beef are displayed to the right of the reference line L1 in the cooking process area 332-2. A cooking video demonstrating how to stir is also displayed to the right of the instructions. Above the instructions, an icon is displayed indicating that the cooking process for Step 4 takes 70 seconds and that 1 second has elapsed since the start of the cooking process. This information is displayed in a fixed position. Information indicating the content of the current cooking process is presented as needed using audio from speaker 32.

[0311] In the example shown in Figure 30, the next cooking step, step 5, is the task of adding seasoning A and stirring. Instructions for adding seasoning A and stirring are displayed in section T3 of the cooking process area 332-1. Above the instructions, an icon is displayed indicating that the cooking time for step 5 is 30 seconds. This information is displayed as it moves to the left.

[0312] The process number area 333 is a display area for information indicating the progress of the cooking process. In the example in Figure 30, it is shown using the cooking process number that step 3 has been completed and the current cooking process is step 4. Among all the cooking processes, the numbers of cooking processes that require particular attention are highlighted using underlines or other means. For example, the numbers of cooking processes with complex operations are highlighted.

[0313] As described above, the cooking navigation screen displays information such as the tasks involved in the current cooking process, a video corresponding to the current cooking process, and the time and elapsed time of the current cooking process. The tasks involved in each cooking process are presented using at least one of the following: text, audio guidance, a graph showing time-series data, or a video taken during the recorded cooking process. The cooking navigation screen also displays information such as the tasks involved in the next cooking process, the progress of all processes, sensor data recorded in the navigation data, and sensor data detected during the reproduction cooking process.

[0314] From the cooking navigation screen, the recreated cook can check the details of the current cooking process and, while performing the instructed tasks, check the details of the next cooking process. In addition, the recreated cook can check how much their work deviates from the record cook's work by looking at the real-time updated graph G11 display.

[0315] • Diagram 32, showing the transition of the cooking navigation screen, is an example of the cooking navigation screen after a predetermined time has elapsed. During the reproduction cooking process, the display of the cooking navigation screen shown in Figure 30 changes to the state shown in Figure 32. The cooking navigation screen shown in Figure 32 is the screen displayed when the current cooking process is still at step 4.

[0316] In the example shown in Figure 32, the information for the current cooking process, step 4, is displayed in the cooking process area 332-2 at the same position as in Figure 30. The elapsed time since the start of the cooking process is 22 seconds. The cooking navigation screen shown in Figure 32 is the screen displayed 21 seconds after the state shown in Figure 30.

[0317] On the other hand, information for the next cooking step, step 5, is displayed within the cooking process area 332-1, at a position closer to the reference line L1 indicating the current time than the position shown in Figure 30. In the example in Figure 32, section T3, which is the section for step 5, is displayed extending to the left of the cooking navigation screen.

[0318] Figure 33 shows an example of the cooking navigation screen after a predetermined amount of time has elapsed. The cooking navigation screen shown in Figure 33 is the screen displayed after the current cooking process has switched to step 5. After more time has passed since the state shown in Figure 32, and step 4 has finished, the current cooking process switches to step 5.

[0319] In the example shown in Figure 33, the information for step 5, which had previously been displayed in a shifted manner, is now fixed and displayed as new information for the current cooking process, located to the right of the reference line L1 within the cooking process area 332-1. The display of the reference line L1 has also switched from being displayed across the time-series data area 331 and the cooking process area 332-2 to being displayed across the time-series data area 331 and the cooking process area 332-1.

[0320] Furthermore, in the example shown in Figure 33, information indicating the content of the next cooking step, step 6, is displayed in section T4 of the cooking process area 332-2. The information indicating the content of step 6 is displayed by gradually moving to the left as time progresses, so that the instructions become clearer.

[0321] Figure 34 shows the transitions between the cooking navigation screens as described above.

[0322] The upper, middle, and lower sections of Figure 34 show the cooking navigation screens from Figures 30, 32, and 33, respectively. In the examples in the upper and middle sections of Figure 34, the information for the current cooking step, Step 4, is displayed in a fixed position, while the information for the next cooking step, Step 5, is displayed moving to the left, approaching the current time, which serves as the reference point.

[0323] In the lower example of Figure 34, after the current cooking process has switched to step 5, the information for the current cooking process, step 5, is displayed in a fixed position, while the information for the next cooking process, step 6, is displayed moving to the left so as to approach the reference current time.

[0324] In this way, on the cooking navigation screen, the information for the current cooking step is displayed in a fixed position, and the information for the next cooking step moves to the left gradually as time progresses, making the instructions clearer. When switching to the next cooking step, the information that was previously displayed as the current cooking step is fixed in place, and the information for the next cooking step then moves to the left gradually as time progresses, making the instructions clearer.

[0325] The speed at which information for the next cooking step is transferred depends on the time elapsed in the current cooking step. The longer the current cooking step takes, the slower the information for the next step is transferred. Conversely, the shorter the current cooking step takes, the faster the information for the next step is transferred.

[0326] This type of display is achieved using cooking process area 332-1 and cooking process area 332-2. The current cooking process information is displayed fixed in cooking process area 332-1 and then fixed in cooking process area 332-2, switching alternately each time the current cooking process changes. In addition, the next cooking process information is displayed by moving within cooking process area 332-1 and then moved within cooking process area 332-2, switching alternately each time the current cooking process changes.

[0327] This transition in the cooking navigation screen allows the simulated chef to intuitively understand when the next cooking step is imminent. Furthermore, the simulated chef can prepare for the next cooking step while simultaneously performing the current one.

[0328] Regarding the switching of cooking processes, the target state that serves as the condition for switching cooking processes is set using the TTW value, etc. The conditions for switching may also be set using sensor data other than the TTW value, such as water evaporation rate, sound, vibration, acceleration, and images. The conditions may also be set by combining multiple types of sensor data, rather than using only one type of sensor data. When conditions combining multiple types of sensor data are set, multiple graphs showing the time-series data of each sensor data are displayed in the time-series data area 331. Alternatively, one graph showing the time-series data of one sensor data of interest may be displayed in the time-series data area 331.

[0329] The weight of the ingredients may be measured before the ingredients are added, or it may be measured in the cooking system 11 during the re-cooking process (after they have been added). If the ingredients are measured during the re-cooking process, information indicating the current weight of the ingredients will be displayed as part of the information regarding the current cooking process, in addition to the instructional text and video.

[0330] The cooking process is switched when the sensor data detected during the reproduction cooking process meets the pre-set target conditions. If the target conditions are set using temperature or moisture evaporation, the cooking process is switched when the temperature or moisture evaporation detected during the reproduction cooking process reaches the set value. In addition, the cooking process is switched when the food temperature reaches a certain temperature and the moisture evaporation after a predetermined time reaches the set value.

[0331] The time-series data of sensor data detected during the reproduction cooking process may be compared with the time-series data of sensor data recorded in the navigation data, and the speed at which information for the next cooking step is transferred may be adjusted based on the comparison results of the time-series data. For example, if the changes in temperature or moisture evaporation during the reproduction cooking process are small, the speed at which information for the next cooking step is transferred will be slower, and if the changes are large, the speed at which information for the next cooking step is transferred will be faster.

[0332] The cooking process may be switched when the amount of water evaporation detected after the recreated cook has performed the stirring operation reaches the target amount of water evaporation set. In this case, for example, whether or not the recreated cook performed stirring during the recreated cooking is detected by the recreated processing unit 11B based on the image captured by the camera 33.

[0333] If stirring has not been performed when the amount of water evaporation detected during the reproduction cooking process reaches a predetermined threshold, such as 2 / 3 of the target amount of water evaporation, the cook will be given supplementary information instructing them to begin stirring. The conditions for the target state are set using the number of stirrings and the amount of water evaporation, and if it is difficult to perform the predetermined number of stirrings in the remaining time, adjustments such as adjusting the heat may be made to reduce the rate of water evaporation.

[0334] Such automatic adjustments, which reflect the state of the work, are performed only for conditions that have little impact on the reproducibility of the finished dish. In this example, the number of stirring cycles and the amount of water evaporated are conditions that have a large impact on reproducibility, while the rate of water evaporation is a condition that has little impact on reproducibility.

[0335] For tasks where timing differences are likely to occur, such as adding ingredients, the cooking process may be switched when a button displayed on the cooking navigation screen is pressed.

[0336] Regarding the display of time-series data, sensor data recorded in the navigation data and sensor data detected during the reproduction cooking process are presented to the cook as time-series data using graphs and other methods. The graph showing the sensor data recorded in the navigation data and the graph showing the sensor data detected during the reproduction cooking process are displayed synchronously, starting from the beginning of the cooking process.

[0337] The information displayed using graphs, etc., is time-series data obtained by converting information acquired by a group of sensors installed in the cooking system 11 and cooking utensils into temperature, wavenumber (wavelength, frequency), amount of water evaporation, weight, mixing strength, mixing speed, estimated amount of chemical change (Maillard reaction, etc.), heat transfer simulation values, etc.

[0338] By presenting the sensor data recorded in the navigation data and the sensor data detected during the reproduction cooking process to the chef in a comparable format, the chef can be made aware of the need to minimize the difference between them during their work. This makes it possible to improve the accuracy of the reproduction of the dish. Furthermore, by confirming that there is no difference between the sensor data recorded in the navigation data and the reproduction data, the accuracy of the work can be ensured.

[0339] Feedback may be provided to the cook recreating the dish based on the difference between the sensor data recorded in the navigation data and the sensor data detected during the recreation cooking process. For example, if the cook recreating the dish is stirring at a slower rate than the average stirring rate during the recorded cooking process, supplementary information such as "Please stir faster" may be provided to the cook. If the information presented in a graph or similar format represents the stirring speed, the average stirring rate during the recorded cooking process may be presented as a target value, and a warning may be displayed, such as by turning a portion of the screen red, as the difference between the target value and the average stirring rate during the recreation cooking process increases.

[0340] Furthermore, if the current stirring speed of the recreated cook is very close to the standard speed (1-2%) compared to the average stirring speed during the recorded cooking process, feedback such as "Keep it up!" is presented to the recreated cook using a speech bubble or voice message. Such voice messages common to multiple cooking processes are registered, for example, using the recipe editor screen shown in Figure 24. Alternatively, the voice messages may be presented using voices that are standardly stored in the cooking system 11.

[0341] Figure 35 shows an example of how supplementary information is displayed. Supplementary information is information that supplements the work. In addition to being displayed on the cooking navigation screen, supplementary information is also presented using audio as needed.

[0342] In the example shown in Figure 35, a speech bubble 351 containing the words "Please stop occasionally" is displayed, with the time-series data of the current cooking process as the source. Speech bubble 351 is displayed in a position that does not overlap with the information of the current cooking process or the information of the next cooking process.

[0343] During the cooking process, the cook performs the necessary tasks by referring to the text, cooking videos, and audio guides displayed on the cooking navigation screen. However, there are times when supplementary explanations are needed, such as "temporarily stop stirring" or "prepare the next ingredient." In such situations, the display shown in Figure 35 allows for timely and appropriate supplementary explanations regarding the necessary steps.

[0344] As supplementary information, the system may also display information indicating that the process is proceeding as planned, such as "Wait until heating is complete," even though no action is required from the user. This supplementary information explaining the waiting period may be displayed overlaid on the information indicating the current cooking process.

[0345] These supplementary information messages are displayed when sensor data detected during the cooking process meets pre-set conditions. For example, when the elapsed time of a certain cooking step reaches 60 seconds before the scheduled time to switch to the next cooking step, the message "Prepare the next ingredient" is displayed. Also, when the temperature of an ingredient detected in a certain cooking step exceeds a threshold temperature, the message "Continue stirring" is displayed, and when the stirring process is complete, the message "Temporarily stop stirring" is displayed.

[0346] The conditions used to determine whether to display supplemental data are set using parameters such as time, wavenumber (wavelength, frequency), temperature, water evaporation rate, weight, stirring intensity, stirring direction, stirring speed, estimated chemical change (e.g., Maillard reaction), and heat transfer simulation values. Conditions may be set using a combination of multiple parameters, not just a single parameter.

[0347] <<Modification>> Although navigation data containing multiple types of data with different reproduction levels is generated, it is also possible to generate navigation data containing only one type of data corresponding to the reproduction level of the reproduction chef. In this case, the information processing unit 12A of the information processing server 12 identifies the reproduction level of the reproduction chef based on the information transmitted from the reproduction processing unit 11B, and generates navigation data containing data corresponding to the identified reproduction level. The information processing unit 12A transmits the generated navigation data to the reproduction processing unit 11B to perform the reproduction cooking.

[0348] The cooking-related information used to generate structured data does not necessarily have to be information obtained using the cooking system 11 shown in Figure 3. For example, a simple device could be attached to a home induction cooktop to acquire temperature history and time-series data of heat settings. The information acquired by the device is uploaded to the information processing server 12 and used as cooking-related information to generate navigation data.

[0349] In this case, structured data may be generated by supplementing missing recognition data, equipment operation data, sensor data, etc., from past pre-cooking information and cooking-related information stored on the information processing server 12. This reduces the effort required for manual correction when generating navigation data. Multiple people may also be able to simultaneously correct and edit the cooking process on the information processing server 12.

[0350] • Diagram 36, a modified example of the system configuration, shows an example of the configuration of the cooking support system 1.

[0351] In the cooking support system 1 shown in Figure 36, the processes of recording cooking, generating navigation data, and reproducing cooking are all performed within the same cooking system 11. In this way, it is possible to generate navigation data using other information processing devices, such as a standalone computer installed in the kitchen where the induction cooktop 21 is located.

[0352] As shown in Figure 37, recording and reproduction cooking may be performed in different cooking systems 11. In the example shown in Figure 37, the recording processing unit 11A is implemented in cooking system 11-1, and the reproduction processing unit 11B is implemented in cooking system 11-2. Cooking systems 11-1 and 11-2 each have the same configuration as cooking system 11 described with reference to Figure 3, etc.

[0353] The recording data generated by the recording processing unit 11A of the cooking system 11-1 is transmitted to the information processing server 12 via a network such as the Internet, and is used by the information processing unit 12A to generate a navigation dataset. The navigation data created by the information processing unit 12A is transmitted to the cooking system 11-2 via the network, and is used by the reproduction processing unit 11B for reproduction cooking.

[0354] By providing multiple cooking systems 11-2 that perform reproduction cooking, it becomes possible to reproduce a dish based on a single navigation data in various locations.

[0355] Thus, it is possible to arbitrarily change which device performs at least one of the following processes: recording the cooking process, generating navigation data, or reproducing the cooking process.

[0356] • Example of a computer configuration: The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on the computer. Here, a computer includes computers that are built into dedicated hardware, as well as general-purpose personal computers that can perform various functions by installing various programs.

[0357] Figure 38 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program. The information processing server 12 also has a configuration similar to that shown in Figure 38.

[0358] In a computer, the processing circuit 401, ROM (Read Only Memory) 402, and RAM (Random Access Memory) 403 are interconnected by a bus 404.

[0359] An input / output interface 405 is further connected to the bus 404. An input unit 406, an output unit 407, a storage unit 408, a communication unit 409, and a drive 410 are connected to the input / output interface 405.

[0360] The input unit 406 may include physical or virtual means of operation that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means of inputting information by the user through voice or gaze. Furthermore, the input unit 406 may include sensors for inputting various physical quantities to the computer. For example, the input unit 406 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 406 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, and distance. The output unit 407 may include means of presenting information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 408 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various types of information (including programs). The communication unit 409 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 410 drives removable media 411 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0361] The processing circuit 401 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 401 (its processor) performs the above-described series of processes by loading the program stored in the storage unit 408 into the RAM 403 via the input / output interface 405 and the bus 404 and executing it. The processing circuit 401 can output the processing results of the series of processes from the output unit 407, for example, via the bus 404 and the input / output interface 405, as needed. The processing circuit 401 can also store the processing results in the storage unit 408 or transmit them from the communication unit 409.

[0362] The program executed by the computer (processing circuit 401) can be provided by recording it on a removable medium 411, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0363] In a computer, a program can be installed in the storage unit 408 via the input / output interface 405 by inserting the removable media 411 into the drive 410. Alternatively, a program can be received by the communication unit 409 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 408. Furthermore, programs can be pre-installed in the ROM 402 or the storage unit 408.

[0364] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0365] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0366] The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there.

[0367] In this specification, a system means one or more components (devices, modules, etc.). Therefore, one or more components of a computer, for example, only the processor, or a combination of a processor and memory, constitute a system. Regarding a set of multiple components, it is not necessary whether all components reside in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, or a single device containing multiple modules within a single enclosure, are all systems. Furthermore, for example, an entire computer, or a combination of a computer and other devices such as a server (not shown), also constitutes a system.

[0368] The embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0369] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network. Furthermore, each step described in the flowchart above can be executed by a single device or shared among multiple devices. Additionally, if a single step includes multiple processes, these processes can be executed by a single device or shared among multiple devices.

[0370] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0371] <Examples of configuration combinations> This technology can also be configured as follows:

[0372] (1) An information processing device comprising: an acquisition unit that acquires recording data that records the state of cooking performed by a cook; and a generation unit that generates data based on the recording data, which includes information about tasks according to their difficulty level, to be used as navigation data to present the content of each step of the cooking process when the recorded cooking is reproduced. (2) The information processing device according to (1), wherein the generation unit generates the navigation data which includes information that expresses the tasks of a predetermined process as different tasks according to their difficulty level. (3) The information processing device according to (2), wherein the generation unit generates the navigation data which includes multiple types of information with different difficulty levels. (4) The information processing device according to (2) or (3), wherein the generation unit generates the navigation data which includes information that expresses the tasks of a predetermined process as tasks that take longer than the time it takes for the cook to perform them. (5) The information processing device according to any one of (1) to (4), wherein the generation unit generates the navigation data which presents the content of the tasks of a predetermined process using types of information according to their difficulty level. (6) The information processing device according to any one of (1) to (5) above, wherein the recorded data includes time-series data indicating the state at each timing during cooking. (7) The information processing device according to (6) above, wherein the recorded data includes recognition data which is time-series data indicating at least one of the ingredients used by the cook, the cooking utensils used by the cook, and the type of work performed by the cook, as recognized based on images taken during cooking. (8) The information processing device according to (6) or (7) above, wherein the recorded data includes equipment operation data which is time-series data relating to the operation of cooking equipment by the cook, measured during cooking. (9) The information processing device according to any one of (6) to (8) above, wherein the recorded data includes sensor data which is at least one of the time-series data indicating changes in the state of ingredients during cooking and time-series data indicating changes in the state of cooking utensils. (10) The information processing device according to any one of (6) to (9) above, further comprising a data conversion unit that converts the recorded data into structured data which associates the time-series data in intervals divided at timings in which a predetermined change has occurred in the time-series data.(11) The information processing device according to (10), further comprising a process determination unit that determines the timing of the division of each cooking process performed by the cook based on the structured data. (12) The information processing device according to (11), wherein the time-series data includes recognition data indicating the type of work performed by the cook, recognized based on images taken during cooking, and the process determination unit determines the timing of the division of each process based on the type of work performed by the cook in each section. (13) The information processing device according to (12), wherein the process determination unit determines sections in which the type of work performed by the cook satisfies predetermined conditions as sections of the same process. (14) The information processing device according to (13), further comprising a correction unit that corrects the timing of the division of each process determined by the process determination unit in accordance with user operations. (15) The information processing device according to any one of (1) to (14), further comprising a correction unit that corrects the navigation data generated by the generation unit in accordance with user operations. (16) The information processing device according to (6), wherein the generation unit generates navigation data that presents the content of the work of each process using at least one of the following: text, sound, a graph showing the time-series data, and images taken while the cook is cooking. (17) The information processing device according to any one of (1) to (16), further comprising a management unit that provides a plurality of the navigation data generated based on different recorded data to a cooking system that reproduces the cooking that was the subject of recording. (18) An information processing method comprising: an information processing device acquiring recorded data that records the state of cooking by a cook; and generating data based on the recorded data that includes information about the work according to its difficulty level as navigation data used to present the content of the work of each process when the cooking that was the subject of recording is reproduced. (19) A program for causing a computer to execute a process that includes: acquiring recorded data that records the state of cooking by a cook; and generating data based on the recorded data that includes information about the work according to its difficulty level as navigation data used to present the content of the work of each process when the cooking that was the subject of recording is reproduced.

[0373] 1 Cooking support system, 11 Cooking system, 11A Recording processing unit, 11B Reproduction processing unit, 12 Information processing server, 12A Information processing unit, 21 IH cooker, 31 Display, 32 Speaker, 33 Camera, 34 Microphone, 35 Lighting equipment, 36 Cooking environment measuring instrument, 37 Router, 101 Editing processing unit, 121 Recording data acquisition unit, 122 Data conversion unit, 123 Process discrimination unit, 124 Process correction unit, 125 Basic information correction unit, 126 Navigation data generation unit, 127 Navigation data correction unit, 128 Navigation data storage unit

Claims

1. An information processing device comprising: an acquisition unit that acquires recording data that records the state of cooking performed by a cook; and a generation unit that generates data based on the recording data, which contains information about the tasks according to their difficulty level, to be used as navigation data to present the content of each step of the cooking process when the recorded cooking is reproduced.

2. The information processing apparatus according to claim 1, wherein the generation unit generates navigation data that includes information representing the work of a predetermined process as different work according to its difficulty level.

3. The information processing apparatus according to claim 2, wherein the generation unit generates the navigation data which includes multiple types of information with different difficulty levels.

4. The information processing apparatus according to claim 2, wherein the generation unit generates navigation data that includes information expressing the work of a predetermined process as a work that takes longer than the time it takes for the cook to perform the work.

5. The information processing apparatus according to claim 1, wherein the generation unit generates navigation data that presents the content of work in a predetermined process using information of a type corresponding to the difficulty level.

6. The information processing apparatus according to claim 1, wherein the recorded data includes time-series data indicating the state at each timing during cooking.

7. The information processing apparatus according to claim 6, wherein the recorded data includes recognition data which is time-series data indicating at least one of the ingredients used by the cook, the cooking utensils used by the cook, and the type of work performed by the cook, based on images taken during cooking.

8. The information processing apparatus according to claim 6, wherein the recorded data includes equipment operation data, which is time-series data of information relating to the operation of cooking equipment by the cook measured during cooking.

9. The information processing apparatus according to claim 6, wherein the recorded data includes sensor data which is at least one of the time-series data showing changes in the state of ingredients during cooking and time-series data showing changes in the state of cooking utensils.

10. The information processing apparatus according to claim 6, further comprising a data conversion unit that converts the recorded data into structured data in which the time series data of intervals divided at the timing of a predetermined change in the time series data is associated.

11. The information processing apparatus according to claim 10, further comprising a process determination unit that determines the timing of the division of each cooking process performed by the cook based on the structured data.

12. The information processing apparatus according to claim 11, wherein the time-series data includes recognition data indicating the type of work performed by the cook, based on images taken during cooking, and the process discrimination unit determines the timing of the end of each process based on the type of work performed by the cook in each section.

13. The information processing apparatus according to claim 12, wherein the process determination unit determines that sections in which the type of work performed by the cook satisfies predetermined conditions are sections of the same process.

14. The information processing apparatus according to claim 13, further comprising a correction unit that corrects the timing of the end of each process determined by the process determination unit in accordance with user operation.

15. The information processing apparatus according to claim 1, further comprising a correction unit for correcting the navigation data generated by the generation unit in accordance with user operations.

16. The information processing apparatus according to claim 6, wherein the generation unit generates navigation data that presents the content of the work of each process using at least one of the following: text, audio, a graph showing the time-series data, and images taken by the cook while cooking.

17. The information processing apparatus according to claim 1, further comprising a management unit that provides a plurality of navigation data generated based on different recorded data to a cooking system in which the recorded cooking is reproduced.

18. An information processing method comprising: an information processing device acquiring recorded data that records the state of cooking performed by a cook; and generating data based on the recorded data that contains information about the tasks according to their difficulty level, to be used as navigation data to present the content of each step of the cooking process when the recorded cooking is reproduced.

19. A program for causing a computer to perform a process that includes acquiring recording data that records the state of cooking performed by a cook, and generating data based on the recording data that contains information about the tasks according to their difficulty level, to be used as navigation data to present the content of each step of the cooking process when the recorded cooking is reproduced.

Images