Cooking apparatus and method for controlling same

Abstract

Disclosed is a cooking apparatus. This cooking apparatus carries out the following: acquires a first image group including a plurality of thermal images that are acquired by a thermal imaging camera, and acquires a second image group including a plurality of optical images that are acquired by an optical camera; identifies a plurality of first feature points in the first image group and identifies a plurality of second feature points in the second image group; identifies a plurality of first matching points on the basis of the plurality of first feature points for a first thermal image in the first image group and the plurality of second feature points for a first optical image in the second image group; identifies a plurality of second matching points which are the plurality of first matching points plus a plurality of matching points identified on the basis of a plurality of first feature points for a second thermal image in the first image group and a plurality of second feature points for a second optical image in the second image group; identifies a plurality of regions including at least one matching point on the basis of the second matching points and identifies a target matching point for each region; and performs image registration between the image captured by the thermal imaging camera and the image captured by the optical camera on the basis of the target matching point.

Description

Cooking device and control method thereof

[0001] The present invention relates to a cooking device and a method for controlling the same, and more specifically, to a cooking device that performs image registration and a method for controlling the same.

[0002] Recently, technology for AI-powered cooking devices has been advancing. Accordingly, cooking devices equipped with various sensors are being developed to more accurately monitor and utilize the state and changes of food being cooked.

[0003] Various sensors, including optical cameras and thermal cameras, can be mounted at various locations on a cooking device. However, camera images taken at different positions or angles are obtained in different coordinate systems, so it is necessary to transform the images of the same object into images in a single coordinate system. The method of transforming images taken at different positions or angles to appear in a single coordinate system is called image registration.

[0004] A cooking device according to one or more embodiments of the present disclosure comprises a thermal imaging camera positioned to photograph the interior of the cooking device, an optical camera positioned to photograph the interior of the cooking device at a certain distance from the thermal imaging camera, a memory for storing instructions, and one or more processors including processing circuitry.

[0005] According to one or more embodiments, when the instructions are executed individually or collectively, the cooking device acquires a first image group comprising a plurality of thermal images acquired through the thermal imaging camera and acquires a second image group comprising a plurality of optical images acquired through the optical camera; identifies a plurality of first feature points in the first image group and identifies a plurality of second feature points in the second image group; identifies a plurality of first matching points based on the plurality of first feature points for a first thermal image in the first image group and the plurality of second feature points for a first optical image in the second image group; identifies a plurality of second matching points by adding a plurality of matching points identified based on the plurality of first feature points for a second thermal image in the first image group and the plurality of second feature points for a second optical image in the second image group to the plurality of first matching points; and identifies a plurality of regions including at least one matching point based on the second matching points to identify a target matching point for each region. Based on the above target matching point, image registration is performed between the image captured through the thermal imaging camera and the image captured through the optical camera.

[0006] According to one or more embodiments, when the instructions are executed individually or collectively by the one or more processors, the cooking device compares first information corresponding to each of the plurality of first feature points and second information corresponding to each of the plurality of second feature points to identify a matching score between feature points, and identifies a first feature point and a second feature point whose identified matching score exceeds a preset first score as the matching points, wherein the first information is information based on the difference in pixel values ​​between each of the plurality of first feature points and an adjacent pixel, and the second information is information based on the difference in pixel values ​​between each of the plurality of second feature points and an adjacent pixel.

[0007] According to one or more embodiments, the matching point includes information regarding a first coordinate corresponding to a first feature point and a second coordinate corresponding to a second feature point, and when the instructions are executed individually or collectively by the one or more processors, the cooking device identifies a mathematical model including rotation information and movement information for an image captured through the optical camera based on an image captured through the thermal imaging camera, based on the first coordinate and the second coordinate corresponding to the target matching point, and performs the image registration between the image captured through the thermal imaging camera and the image captured through the optical camera based on the mathematical model.

[0008] According to one or more embodiments, when the instructions are executed individually or collectively by one or more processors, the cooking device inputs other matching points among the matching points, excluding the target matching point, into the identified mathematical model to identify error information, and if the error information is greater than or equal to a preset threshold value, identifies a first feature point and a second feature point as the matching points, wherein the identified matching score exceeds a preset second score, and the second score is a score lower than the first score.

[0009] According to one or more embodiments, when the instructions are executed individually or collectively by one or more processors, the cooking device identifies error information by inputting other matching points among the matching points, excluding the target matching point, into the identified mathematical model; if the error information is less than the preset threshold value, the target matching point for each region among the matching points is re-identified; the mathematical model is re-identified based on the re-identified target matching point; and other matching points among the matching points, excluding the re-identified target matching point, into the re-identified mathematical model to identify error information.

[0010] According to one or more embodiments, when the instructions are executed individually or collectively by the one or more processors, the cooking device identifies at least one second matching point located within a predetermined distance based on a first matching point among the matching points, and identifies the matching point group including the third matching point and the fourth matching point.

[0011] According to one or more embodiments, when the instructions are executed individually or collectively by the one or more processors, the cooking device segments an image captured by the thermal imaging camera into a plurality of regions including each of the matching point groups, identifies the target matching point for each region based on the matching score between the feature points and the pixel response intensity, and performs the image registration based on the target matching point, wherein the pixel response intensity is the average value of the brightness difference for each of the plurality of pixels adjacent to the matching point.

[0012] According to one or more embodiments, when the instructions are executed individually or collectively by the one or more processors, the cooking device identifies a pixel in which the difference in pixel values ​​corresponding to each adjacent pixel among a plurality of pixels included in an image captured through the thermal imaging camera exceeds a preset range as the first feature point, and identifies a pixel in which the difference in pixel values ​​corresponding to each adjacent pixel among a plurality of pixels included in an image captured through the optical camera exceeds a preset range as the second feature point.

[0013] According to one or more embodiments, when the instructions are executed individually or collectively by the one or more processors, the cooking device identifies at least one of the position information and shooting direction of each of the thermal imaging camera and the optical camera, and acquires the second image converted based on the position information of each of the thermal imaging camera and the optical camera and the shooting direction of the thermal imaging camera, an image acquired through the optical camera.

[0014] A method for controlling a cooking device according to one or more embodiments of the present disclosure comprises: acquiring a first image group including a plurality of thermal images acquired through a thermal imaging camera and acquiring a second image group including a plurality of optical images acquired through an optical camera; identifying a plurality of first feature points in the first image group; identifying a plurality of second feature points in the second image group; identifying a plurality of first matching points based on the plurality of first feature points for a first thermal image in the first image group and the plurality of second feature points for a first optical image in the second image group; and identifying a plurality of second matching points by adding a plurality of matching points identified based on the plurality of first feature points for a second thermal image in the first image group and the plurality of second feature points for a second optical image in the second image group to the plurality of first matching points.

[0015] A non-transient computer-readable storage medium storing computer instructions that cause the cooking device to perform an operation when executed by a processor of a cooking device according to one or more embodiments of the present disclosure, wherein the operation comprises: acquiring a first image group including a plurality of thermal images acquired through a thermal imaging camera and acquiring a second image group including a plurality of optical images acquired through an optical camera; identifying a plurality of first feature points in the first image group; identifying a plurality of second feature points in the second image group; identifying a plurality of first matching points based on the plurality of first feature points for a first thermal image in the first image group and the plurality of second feature points for a first optical image in the second image group; and identifying a plurality of second matching points by adding to the plurality of first matching points a plurality of matching points identified based on the plurality of first feature points for a second thermal image in the first image group and the plurality of second feature points for a second optical image in the second image group.

[0016] FIG. 1 is a drawing for explaining the operation of a cooking device according to one or more embodiments.

[0017] FIG. 2 is a block diagram illustrating the configuration of a cooking device according to one or more embodiments.

[0018] FIG. 3 is a drawing for explaining the process of acquiring an image of a cooking device according to one or more embodiments.

[0019] FIG. 4 is a drawing for explaining the process of identifying feature points of a cooking device according to one or more embodiments.

[0020] FIGS. 5 and 6 are drawings for explaining the process of identifying matching points of a cooking device according to one or more embodiments.

[0021] FIG. 7 is a diagram illustrating the process of identifying matching point groups of a cooking device according to one or more embodiments.

[0022] FIG. 8 is a diagram illustrating the process of identifying target matching points of a cooking device according to one or more embodiments.

[0023] FIG. 9 is a diagram illustrating the process of identifying a mathematical model of a cooking device according to one or more embodiments.

[0024] FIG. 10 is a diagram illustrating a matching point re-identification process based on error information of a mathematical model according to one or more embodiments.

[0025] FIG. 11 is a flowchart illustrating the overall operation of a cooking device according to one or more embodiments.

[0026] The terms used in the various embodiments of this Disclosure have been selected to be as widely used and general as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant description section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0027] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0028] The expression "at least one of A or / and B" should be understood as representing either "A" or "B" or "A and B".

[0029] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0030] Where it is stated that a component (e.g., Component 1) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., Component 2), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., Component 3).

[0031] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this disclosure, terms such as “comprising” or “consisting of” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0032] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor (not shown), except for a "module" or "part" that needs to be implemented in specific hardware.

[0033] In the present disclosure, the term "user" may refer to a person using a cooking device or a device used by such person.

[0034] An embodiment of the present disclosure will be described in more detail below with reference to the attached drawings.

[0035] FIG. 1 is a drawing for explaining the operation of a cooking device according to one or more embodiments.

[0036] The cooking device (100) can perform various cooking processes by applying high-temperature heat generated using electricity or gas to the food being cooked. The cooking device (100) may be a device for cooking food using ingredients prepared by the user. The cooking device (100) may be a microwave oven, an oven, an induction cooktop, a gas range, an air fryer, etc. For convenience of explanation in this disclosure, the cooking device (100) will be described as a microwave oven.

[0037] According to one embodiment, the cooking device (100) can monitor the condition of the food being cooked through a plurality of cameras. The cooking device (100) may have a thermal imaging camera (110) and an optical camera (120) placed at specific locations inside the cooking chamber. The cooking device (100) can photograph the food being cooked through the thermal imaging camera (110) and the optical camera (120) and monitor the condition of the food being cooked based on the captured images.

[0038] For example, the cooking device (100) can identify whether the food is cooked uniformly and / or the degree of cooking over time through a thermal imaging camera (110). For example, the cooking device (100) can identify the degree of burntness of the food over time from an image of the food obtained through the thermal imaging camera (110).

[0039] For example, the cooking device (100) can identify the condition of the food through an optical camera (120). For example, the cooking device (100) can identify the appearance of the food, changes in the color of the food, and whether cooking is complete from an image obtained through the optical camera (120).

[0040] Referring to FIG. 1, the cooking device (100) may include a thermal imaging camera (110) positioned at the top center inside the cooking chamber and an optical camera (120) positioned on the side or top inside the cooking chamber. The thermal imaging camera (110) and the optical camera (120) may be positioned at any point (or the same location) inside the cooking chamber, but may also be positioned separated by a certain distance. In this disclosure, the description assumes a case where the thermal imaging camera (110) and the optical camera (120) are positioned separated by a certain distance to photograph the inside of the cooking chamber.

[0041] According to one embodiment, the cooking device (100) can acquire an image of the inside of the cooking chamber through a thermal imaging camera (110) and an optical camera (120), respectively. The image acquired through the thermal imaging camera (110) and the image acquired through the optical camera (120) are images of the same object or space, but the shooting direction may differ depending on the position of each camera (110, 120).

[0042] According to one example, the image obtained may be an image including a circular turntable located inside a cooking chamber. The turntable may be a circular rotating plate for rotating the food so that the food can be cooked uniformly.

[0043] Referring to FIG. 1, the cooking device (100) can obtain an image (10) visualizing the heat distribution inside the cooking chamber through a thermal imaging camera (110). The image (10) obtained through the thermal imaging camera (110) may be a ring-shaped heat distribution image due to a turntable located inside the cooking chamber. The cooking device (100) can obtain a captured image (20) that allows visual inspection of the inside of the cooking chamber through an optical camera (120). The image (20) obtained through the optical camera (120) may include an image of the turntable inside the cooking chamber.

[0044] According to one embodiment, the cooking device (100) can perform image registration to match an image (10) obtained through a thermal imaging camera (110) and an image (20) obtained through an optical camera (120) into a single image.

[0045] Image registration may include an operation for aligning or matching two or more images captured according to different directions, positions, scales, and angles into a single image. For example, image registration may include an operation for aligning or matching an image (10) captured through a thermal imaging camera (110) and an image captured through an optical camera (120) into a single image with the same direction, position, scale, and angle. Image registration is not limited thereto and may be referred to in various ways, such as image alignment, image matching, and image synthesis, but in this disclosure, it will be collectively referred to as image registration.

[0046] However, each image captured through the thermal imaging camera (110) and the optical camera (120) is an image including a circular or ring-shaped image including a turntable, as illustrated in FIG. 1, and there may be difficulties in the process of registering each image as a single image.

[0047] Hereinafter, various embodiments in which a cooking device (100) performs image registration based on each image captured through a thermal imaging camera (110) and an optical camera (120) will be described with reference to the drawings.

[0048] FIG. 2 is a block diagram illustrating the configuration of a cooking device according to one or more embodiments.

[0049] According to FIG. 2, the cooking device (100) includes a thermal imaging camera (110), an optical camera (120), a memory (130), and one or more processors (140). However, it is not limited thereto, and the cooking device (100) may be implemented with some components excluded or with other components included.

[0050] A thermal imaging camera (110) may be a camera for detecting thermal radiation emitted from an object and acquiring an image visualized in various colors according to temperature. The thermal imaging camera (110) may display different colors according to temperature so that the temperature can be identified with the naked eye. The image acquired through the thermal imaging camera (110) may be an image visualizing the heat distribution of a specific object, background, or space.

[0051] The optical camera (120) may be a camera for detecting the color of an object using visible light and acquiring a color image by combining the detected color into R, G, and B colors. The optical camera (120) may include a lens, a shutter, and an image sensor configuration. The image acquired through the optical camera (120) may be a color image that allows a specific object, background, or space to be visually identified.

[0052] The memory (130) can store at least one instruction, data, program, etc. required for the operation of the cooking device (100). For example, the memory (130) can store pixel information included in an image captured through a camera.

[0053] The memory (130) may be implemented in the form of a memory embedded in the cooking device (100) or in the form of a memory detachable from the cooking device (100), depending on the purpose of data storage. For example, data for operating the cooking device (100) may be stored in a memory embedded in the cooking device (100), and data for the expansion function of the cooking device (100) may be stored in a memory detachable from the cooking device (100).

[0054] In the case of memory embedded in the cooking device (100), it may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD).

[0055] The memory (130) may be implemented as a single memory that stores data generated in various operations according to the present disclosure, but is not limited thereto, and the memory (130) may be implemented to include a plurality of memories that each store different types of data or each store data generated in different stages.

[0056] One or more processors (140) control the overall operation of the cooking device (100). Specifically, one or more processors (140) may be connected to each component of the cooking device (100) to control the overall operation of the cooking device (100). For example, one or more processors (140) may be electrically connected to a thermal imaging camera (110), an optical camera (120), and a memory (130) to control the overall operation of the cooking device (100). One or more processors (140) may be composed of one or more processors.

[0057] One or more processors (140) can perform the operation of a cooking device (100) according to various embodiments by executing one or more instructions stored in memory (130).

[0058] One or more processors (140) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. One or more processors (140) may control one or any combination of other components of the cooking device and may perform operations or data processing related to communication. One or more processors (140) may execute one or more programs or instructions stored in memory. For example, one or more processors may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in memory.

[0059] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0060] One or more processors (140) may be implemented as a single-core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When one or more processors (140) are implemented as multicore processors, each of the multiple cores included in the multicore processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by multiple cores may be included in the multicore processor. Additionally, each of the multiple cores included in the multicore processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one or more embodiments of the present disclosure.

[0061] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0062] In the embodiments of the present disclosure, a processor may refer to a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. For convenience of explanation, one or more processors (140) will be referred to as processors (140) below.

[0063] According to one embodiment, the processor (140) may acquire a first image group including a plurality of thermal images acquired through a thermal imaging camera (110) and acquire a second image group including a plurality of optical images through an optical camera (120).

[0064] According to one embodiment, the processor (140) can identify a plurality of first feature points in a first image group and a plurality of second feature points in a second image group.

[0065] Pixel information may include information on at least one of color information corresponding to R, G, and B subpixels, location information of the corresponding pixel within the image, and brightness information based on the color information.

[0066] A feature point may be any one of multiple pixels that are distinguished from other pixels due to a difference in color value with surrounding pixels within an image. For example, if the brightness value of a specific pixel among multiple pixels is greater than or equal to a preset value than the brightness value of surrounding pixels, that pixel can be identified as a feature point. Feature points are not limited to this and may be referred to in various ways, such as key points, feature vectors, or points of interest; however, in this disclosure, they will be collectively referred to as feature points.

[0067] According to one embodiment, the processor (140) can identify a plurality of first matching points based on a plurality of first feature points for a first thermal image among a first image group and a plurality of second feature points for a first optical image among a second image group. A matching point may be a point that includes both a feature point in the first image and a feature point in the second image that corresponds to it. That is, a matching point may include a second feature point corresponding to a first feature point, two points, and coordinate values ​​corresponding to each point.

[0068] According to one embodiment, the processor (140) can identify a plurality of second matching points by adding a plurality of matching points identified based on a plurality of first feature points for a second thermal image among a first image group and a plurality of second feature points for a second optical image among a second image group to a plurality of first matching points.

[0069] According to one example, the first thermal image, the first optical image, the second thermal image, and the second optical image may be images in each frame in chronological order. For example, in a frame interval containing three frames, the first thermal image may be the first frame and the second thermal image may be the second frame.

[0070] According to one embodiment, the processor (140) can identify a plurality of regions including at least one matching point based on a second matching point to identify a target matching point for each region.

[0071] According to one embodiment, the processor (140) can identify at least one matching point group including at least one matching point based on distance information between a plurality of matching points. The matching point group may be a group including all matching points included within a predetermined distance based on a specific matching point.

[0072] According to one embodiment, the processor (140) may identify a plurality of regions including each matching point group in a first image to identify a target matching point for each region. The target matching point may be one matching point arbitrarily identified for each region among the plurality of matching points to identify a mathematical model including rotation information and translation information for the image.

[0073] According to one embodiment, the processor (140) can perform image registration between an image captured through a thermal imaging camera and an image captured through an optical camera based on target matching points.

[0074] FIG. 3 is a drawing for explaining the process of acquiring an image of a cooking device according to one or more embodiments.

[0075] According to one embodiment, the cooking device (100) can acquire a first image (10) through a thermal imaging camera (110) and an optical image (410) through an optical camera (120).

[0076] According to one embodiment, the cooking device (100) identifies at least one of the position information and shooting direction of each of the thermal imaging camera (110) and the optical camera (120), and can obtain a second image (420) converted based on the position information of each of the thermal imaging camera (110) and the optical camera (120) and the shooting direction of the thermal imaging camera (110), and the optical image (410) obtained through the optical camera (120).

[0077] A thermal imaging camera (110) and an optical camera (120) within a cooking device (100) may be positioned so as to be separated by a certain distance or more to photograph the interior of the cooking device (100). In this case, the shooting direction of the image (10) captured through the thermal imaging camera (110) and the optical image (410) captured through the optical camera (120) may be different. In this case, the cooking device (100) may obtain a second image (420) that is converted based on the shooting direction of the thermal imaging camera (110) from the optical image (410) captured through the optical camera (120).

[0078] According to one example, the cooking device (100) can identify the position information and shooting direction of the thermal imaging camera (110) and the optical camera (120). For example, the cooking device (100) can identify values ​​corresponding to the coordinate values ​​and direction vectors of the thermal imaging camera (110) and the optical camera (120) with the center of the turntable inside the cooking device (100) as the origin. In this case, the cooking device (100) can obtain a second image (420) converted based on the coordinate values ​​and direction vectors of each identified camera from the optical image (410) captured through the optical camera (120).

[0079] Referring to FIG. 3, the cooking device (100) can change the shooting direction of an optical image (410) captured through an optical camera (120) based on the position information and direction vector of a thermal imaging camera (120). For example, when a thermal imaging camera (110) is positioned at the top inside the cooking device (100) and shoots toward the bottom surface, and an optical camera (120) is positioned at the upper side of the cooking device and shoots toward the bottom surface diagonally, the cooking device (100) can obtain a second image (420) that is changed based on the shooting direction of the thermal imaging camera (110) from the optical image (410) captured through the optical camera (120).

[0080] FIG. 4 is a drawing for explaining the process of identifying feature points of a cooking device according to one or more embodiments.

[0081] According to one embodiment, the cooking device (100) can identify a plurality of pixels as first feature points in which the difference between the pixel values ​​corresponding to each adjacent pixel among a plurality of pixels included in a first image (10) exceeds a preset range.

[0082] According to one embodiment, the cooking device (100) can identify a plurality of pixels as second feature points in which the difference between the pixel values ​​corresponding to each adjacent pixel among a plurality of pixels included in a second image exceeds a preset range.

[0083] The pre-set range may be a range that numerically expresses the difference in pixel values ​​corresponding to the R, G, and B subpixels of each pixel. The pre-set range may be set during the manufacturing stage and may be a numerical range for distinguishing a specific pixel as a pixel that exhibits a significantly different color or brightness compared to surrounding pixels among a plurality of pixels.

[0084] In Figure 4, for the convenience of explanation, the brightness or color value corresponding to the R subpixel or the G subpixel is expressed numerically and explained, rather than the brightness or color value corresponding to all R, G, and B subpixels.

[0085] Referring to FIG. 4, the cooking device (100) can identify a pixel (410-5) as a first feature point, wherein the difference in pixel values ​​corresponding to each of the adjacent pixels (410-1 to 410-4, 410-6 to 410-9) among the plurality of pixels (410-1 to 410-n) included in the first image (10) exceeds a preset range (e.g., 5 to 6). In the same manner as described above, the cooking device (100) can identify a second feature point in the second image (420).

[0086] FIGS. 5 and 6 are drawings for explaining the process of identifying matching points of a cooking device according to one or more embodiments.

[0087] According to one embodiment, the cooking device (100) can identify a matching score between feature points by comparing first information corresponding to each of a plurality of first feature points and second information corresponding to each of a plurality of second feature points.

[0088] According to one example, the first information corresponding to each of the first feature points may be information based on the difference in pixel values ​​between each of the plurality of first feature points and an adjacent pixel. For example, it may be data information based on the difference in brightness or color values ​​between each of the first feature points and each adjacent pixel.

[0089] For example, the first information may be binarized data corresponding to the position information and brightness difference between the first feature point and each pixel (first pixel, second pixel, third pixel) when the brightness difference between the first pixel and the adjacent pixels of the first feature point is 3, the brightness difference between the second pixel and the first pixel is 5, and the brightness difference between the third pixel and the first feature point is 2.

[0090] According to one example, the second information corresponding to each of the second feature points may be information based on the difference in pixel values ​​between each of the multiple second feature points and an adjacent pixel.

[0091] Referring to FIG. 5, the cooking device (100) can identify first information (510-1 to 510-3) corresponding to each of a plurality of first feature points (51 to 53). Likewise, the cooking device (100) can identify second information (510-1 to 510-3) corresponding to each of a plurality of second feature points (54 to 56).

[0092] According to one embodiment, the cooking device (100) can identify a matching score between feature points by comparing first information corresponding to a first feature point with second information corresponding to each of a plurality of second feature points.

[0093] The matching score may be a numerical value based on whether the data at each position is identical by comparing the binarized data of the first information and the second information. For example, if only the data at the 4th position of the first information and the second information is different, the matching score may be 90 points. For example, if the data at the 2nd, 3rd, and 4th positions of the first information and the second information is different, the matching score may be 70 points.

[0094] Referring to FIG. 6, the cooking device (100) can identify a matching score (620) between feature points by comparing the first information (510-1) with a plurality of second information (610-1 to 610-6). When comparing the first information (510-1) with the first second information (610-1) among the plurality of second information (610-1 to 610-6), only the data at the 6th position is different, so the matching score between the corresponding feature points may be 90 points.

[0095] According to one embodiment, the cooking device (100) can identify a first feature point and a second feature point as matching points, wherein the identified matching score exceeds a preset first score.

[0096] The first score that has been set may be a value set during the manufacture of the cooking device (100) and / or a value that can be set / changed by the user. Additionally, the first score that has been set may be lowered during the process of re-identifying the matching point.

[0097] According to one example, the cooking device (100) can identify matching points as a set of first feature points and second feature points, where the matching score exceeds a preset first score (e.g., 80 points).

[0098] FIG. 7 is a diagram illustrating the process of identifying matching point groups of a cooking device according to one or more embodiments.

[0099] According to one embodiment, the cooking device (100) can identify at least one second matching point located within a predetermined distance based on a first matching point among a plurality of matching points, and can identify a matching point group including the first matching point and the second matching point.

[0100] The preset distance may be a value set during the manufacture of the cooking device (100) and / or a value that can be set / changed by the user. For example, the preset distance may be 1 cm or 10 cm. For convenience of explanation in this disclosure, the preset distance will be described as 1 cm.

[0101] Referring to FIG. 7, the cooking device (100) can identify a plurality of second matching points (710-2, 710-3) located within a distance of 1 cm (720) from the first matching point (710-1) at the top left among the plurality of matching points. Likewise, the cooking device (100) can identify a plurality of second matching points (710-5, 710-7) located within a distance of 1 cm from the first matching point (710-6) at the bottom right among the plurality of matching points.

[0102] The cooking device (100) can identify a matching point group including a first matching point (710-1) at the top left and a plurality of second matching points (710-2, 710-3). Likewise, the cooking device (100) can identify a matching point group including a first matching point (710-6) at the bottom right and a plurality of second matching points (710-5, 710-7).

[0103] According to one example, the cooking device (100) can identify the first matching point (710-4, 710-8) itself as a matching point group because there are no matching points located within a preset distance for each of the first matching point (710-4, 710-8) at the upper right and the first matching point (710-8) at the lower left.

[0104] FIG. 8 is a diagram illustrating the process of identifying target matching points of a cooking device according to one or more embodiments.

[0105] According to one embodiment, the cooking device (100) may segment a first image into a plurality of regions each comprising a matching point group. The segmentation may include the operation of dividing a plurality of points included in the entire image into one region or one group in pixel units.

[0106] Referring to FIG. 8, the cooking device (100) can segment the first image (10) into a plurality of regions (820-1 to 820-8), each comprising a matching point group (810-1 to 810-4). Each region (820-1 to 820-8) may include at least one of a matching point group (810-1 to 810-4) and a matching point. In FIG. 8, for convenience of explanation, the segmentation is performed into regions of the same size, but it is not limited thereto and the segmentation can be performed into regions of different sizes.

[0107] According to one embodiment, the cooking device (100) can identify target matching points for each region based on the matching score between feature points and pixel response intensity.

[0108] Pixel response intensity may be data that quantifies the average value of the brightness difference for each of a plurality of pixels adjacent to a matching point. For example, if the brightness difference with the first pixel among the plurality of pixels adjacent to the matching point is 3, the brightness difference with the second pixel among the plurality of pixels is 1, and the brightness difference with the third pixel among the plurality of pixels is 2, the pixel response intensity of the matching point may be 3.

[0109] Referring to FIG. 8, the cooking device (100) can identify the matching score and pixel response intensity of each of the plurality of matching points included in each area (820-1 to 820-8). Based on the matching score and pixel response intensity, the cooking device (100) can identify target matching points (830-1 to 830-8) for each area (820-1 to 820-8). For example, the cooking device (100) can identify the matching point with the highest sum of the matching score and pixel response intensity among the matching points located within each area (820-1 to 820-8) as the target matching point (830-1 to 830-8).

[0110] FIG. 9 is a diagram illustrating the process of identifying a mathematical model of a cooking device according to one or more embodiments.

[0111] According to one embodiment, the cooking device (100) can identify a mathematical model (970) including rotation information and movement information for a second image based on a first image, based on the first coordinates of a first feature point corresponding to a target matching point and the second coordinates of a second feature point.

[0112] The mathematical model (970) may include a mathematical algorithm for registering or aligning two images located at different positions into a single image. Specifically, the mathematical model may include mathematical expressions or mathematical algorithms such as linear transformations or non-linear transformations.

[0113] According to one embodiment, the cooking device (100) can identify a mathematical model (970) based on at least two target matching points randomly selected from a plurality of target matching points. However, this is merely one embodiment, and the mathematical model (970) can also be identified based on all identified target matching points.

[0114] Referring to FIG. 9, the cooking device (100) can identify a mathematical model (970) including rotation information and movement information for a second image based on a first image, based on a first target matching point (910) and a second target matching point (920) randomly selected among a plurality of target matching points.

[0115] The cooking device (100) can identify movement information for a second image based on a first image based on first coordinates (930) and second coordinates (940) corresponding to a first target matching point (910). For example, if the first coordinates corresponding to the first target matching point (910) are (3, 3) and the second coordinates are (5, 7), the cooking device (100) can identify movement information for the second image based on the first image corresponding to +2 in the x-axis direction and +2 in the y-axis direction.

[0116] The cooking device (100) can identify rotation information for the second image based on the first image based on the first coordinate (930), the second coordinate (940) corresponding to the first target matching point (910), and the first coordinate (950), the second coordinate (960) corresponding to the second target matching point (920).

[0117] For example, if the first coordinates corresponding to the second target matching point (920) are (1, 2) and the second coordinates are (3, 7), the cooking device (100) can identify rotation information for the second image based on the first image based on the first target matching point (910) and the second target matching point (920).

[0118] According to one embodiment, the cooking device (100) can perform image registration between the first image and the second image based on the identified mathematical model (970).

[0119] FIG. 10 is a diagram illustrating a matching point re-identification process based on error information of a mathematical model according to one or more embodiments.

[0120] According to one embodiment, the cooking device (100) can identify error information by inputting other matching points, excluding the target matching point among a plurality of matching points, into an identified mathematical model.

[0121] Error information may be a numerical value corresponding to the degree of accuracy output by inputting other matching points, excluding the target matching point, into a mathematical model identified based on the target matching point. That is, it may be an error value regarding the result value output by inputting multiple other matching points, excluding the target matching point, into the identified mathematical model.

[0122] For example, the error information may be a numerical value corresponding to the degree of error between the second coordinate of another matching point and the coordinate output when the first coordinate of another matching point is input into an identified mathematical model.

[0123] According to one embodiment, if the error information is greater than or equal to a preset threshold value, the cooking device (100) can identify a first feature point and a second feature point as matching points, wherein the identified matching score exceeds a preset second score.

[0124] The previously set second score may be a value set during the manufacture of the cooking device (100) and / or a value that can be set / changed by the user, and may be a score lower than the previously set first score.

[0125] The pre-set threshold value for error information may be n % (n is a positive integer), but for convenience of explanation in this disclosure, the pre-set threshold value is assumed to be 3% for the explanation.

[0126] Referring to FIG. 10, the cooking device (100) can identify error information (1020) by inputting other matching points (1010-1 to 1010-4), excluding target matching points (830-1 to 830-8), into a mathematical model (970). For example, the cooking device (100) can identify error information by inputting a first coordinate of a matching point (1010-1) into a mathematical model and comparing the output coordinate value with a second coordinate of the matching point (1010-1). Similarly, error information can be identified by performing the same operation for other matching points (1010-2 to 1010-4).

[0127] The cooking device (100) can re-identify matching points based on reset conditions if the identified error information is greater than or equal to a preset threshold value (e.g., 3%). The cooking device (100) can identify a first feature point and a second feature point among a plurality of feature points as matching points, wherein the matching score exceeds the preset second score, based on a second score which is lower than the preset first score.

[0128] For example, if the first score is 90 points, the first feature point and the second feature point among the multiple feature points whose matching score exceeds 80 points can be identified as matching points based on 80 points which is lower than the first score.

[0129] According to one example, the cooking device (100) can re-identify a target matching point based on a plurality of re-identified matching points and re-identify a mathematical model based on the re-identified target matching point.

[0130] According to one embodiment, if the error information is less than a preset threshold value, the cooking device (100) can re-identify the target matching point for each region among the matching points.

[0131] According to one example, the cooking device (100) can re-identify target matching points for re-identifying a mathematical model even when the identified error information is below a preset threshold. For example, by inputting more sample matching points into the identified mathematical model, the identified error information may be greater than or equal to a preset threshold, so that the mathematical model can be re-identified even when the error information is below a preset threshold.

[0132] For example, the cooking device (100) can re-identify a matching point for re-identifying a mathematical model even if the error information based on the first identified mathematical model is below a preset threshold value.

[0133] According to one embodiment, the cooking device (100) can identify a mathematical model based on a re-identified target matching point and identify error information by inputting other matching points, excluding the re-identified target matching point among a plurality of matching points, into the re-identified mathematical model.

[0134] According to one embodiment, the cooking device (100) may terminate the operation to re-identify the mathematical model if the error information of the mathematical model identified repeatedly is less than a preset threshold value than a preset number of times.

[0135] According to one embodiment, the cooking device (100) can acquire a first image and a second image in chronological order through a thermal imaging camera (110) during a first frame interval including a plurality of frames.

[0136] According to one embodiment, the cooking device (100) can acquire a third image and a fourth image in chronological order through an optical camera (120) during a first frame interval including a plurality of frames.

[0137] According to one embodiment, the cooking device (100) can identify a plurality of first feature points in each of the first image and the second image and identify a plurality of second feature points in each of the third image and the fourth image.

[0138] According to one embodiment, the cooking device (100) can identify a plurality of first matching points based on a plurality of first feature points for a first image and a plurality of second feature points for a third image.

[0139] According to one embodiment, the cooking device (100) can identify a plurality of second matching points by adding a plurality of matching points identified based on a plurality of first feature points for a second image and a plurality of second feature points for a fourth image to a plurality of first matching points.

[0140] According to one embodiment, the cooking device (100) can identify at least one matching point group including at least one matching point based on a plurality of second matching points, and identify a plurality of regions including the matching point group to identify a target matching point for each region.

[0141] According to one embodiment, the cooking device (100) can identify the mathematical model described above based on the target matching point. The cooking device (100) can identify an error value by inputting other matching points, excluding the target matching point, into the identified mathematical model.

[0142] According to one embodiment, when the identified error value exceeds a preset threshold value, the cooking device (100) can acquire a first image and a second image in chronological order through a thermal imaging camera (110) during a second frame interval including a plurality of frames.

[0143] Afterwards, the cooking device (100) can identify a matching point by repeating the above-described operation and identify a plurality of regions including the matching point to re-identify the target matching point.

[0144] According to one embodiment, the cooking device (100) can identify a mathematical model based on a re-identified target matching point and identify an error value by inputting other matching points, excluding the re-identified target matching point, into the identified mathematical model.

[0145] According to one embodiment, the cooking device (100) can perform image registration based on an identified mathematical model if the identified error value is less than or equal to a preset threshold value.

[0146] FIG. 11 is a flowchart illustrating the overall operation of a cooking device according to one or more embodiments.

[0147] Referring to FIG. 11, in operation 1110, the cooking device (100) can acquire a first image through a thermal imaging camera (110) and acquire a second image through an optical camera (120).

[0148] In operation 1120, the cooking device (100) can identify a plurality of first feature points corresponding to the first image based on pixel information included in the first image, and identify a plurality of second feature points corresponding to the second image based on pixel information included in the second image.

[0149] In operation 1130, the cooking device (100) can identify a plurality of matching points based on a plurality of first feature points and a plurality of second feature points.

[0150] In operation 1140, the cooking device (100) can identify at least one group of matching points including at least one matching point based on distance information between a plurality of matching points.

[0151] In operation 1150, the cooking device (100) can identify a plurality of regions including each matching point group in the first image and identify a target matching point for each region.

[0152] In operation 1160, the cooking device (100) can perform image registration between the first image and the second image based on target matching points.

[0153] The method for identifying multiple matching points, identifying target matching points for each region, and performing image registration between images has been specifically explained in the various embodiments described above, so a redundant explanation is omitted.

[0154] The control method described in FIG. 11 can be performed by a cooking device (100) having the configuration of FIG. 2 described above, but is not necessarily limited thereto and can be performed by a cooking device having various configurations.

[0155] The various embodiments described above may be implemented as individual embodiments, or at least one embodiment may be combined with one another, either wholly or partially, to be implemented together in a single device.

[0156] According to the various embodiments described above, an image captured by a thermal imaging camera and an image captured by an optical camera are acquired as a single image, and the condition of the food can be identified by utilizing the acquired image.

[0157] Meanwhile, the various embodiments described above may be applied to a product as embodiments alone, but at least some of their contents may be combined with other embodiments of the present disclosure to be implemented together.

[0158] The various embodiments described above may be implemented as software containing instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., a cooking device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. Instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory computer-readable storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

[0159] In addition, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided by being included in a computer program product.

[0160] Specifically, a non-transient readable storage medium or computer program product may be provided that stores computer instructions for performing operations such as acquiring a first image through a thermal imaging camera and acquiring a second image through an optical camera; identifying a plurality of first feature points corresponding to the first image based on pixel information included in the first image and identifying a plurality of second feature points corresponding to the second image based on pixel information included in the second image; identifying a plurality of matching points based on a plurality of first feature points and a plurality of second feature points; identifying a group of at least one matching point including at least one matching point based on distance information between a plurality of matching points; identifying a plurality of regions including each matching point group in the first image to identify a target matching point for each region; and performing image registration between the first image and the second image based on the target matching point.

[0161] Computer program products may be distributed in the form of device-readable storage media (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0162] In addition, computer instructions or programs for performing the control method of a cooking device according to the various embodiments described above may be stored on a non-transitory computer-readable medium. When computer instructions stored on such a non-transitory computer-readable medium are executed by the processor of a specific device, they cause the specific device to perform processing operations according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of a non-transitory computer-readable medium may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, ROMs, etc.

[0163] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In a cooking device, A thermal imaging camera positioned to photograph the interior of the above-mentioned cooking device; An optical camera positioned at a certain distance from the thermal imaging camera to photograph the interior of the cooking device; Memory for storing instructions; and One or more processors including processing circuitry; and The above one or more processors, When the above instructions are executed individually or collectively, the cooking device, A first image group comprising a plurality of thermal images obtained through the thermal imaging camera, and a second image group comprising a plurality of optical images obtained through the optical camera, Identifying a plurality of first feature points in the above first image group and Identifying a plurality of second feature points in the above second image group, and Identifying a plurality of first matching points based on the plurality of first feature points for the first thermal image among the first image group and the plurality of second feature points for the first optical image among the second image group, and Identifying a plurality of second matching points by adding a plurality of matching points identified based on a plurality of first feature points for a second thermal image among a first image group and a plurality of second feature points for a second optical image among a second image group to the plurality of first matching points above, and Based on the second matching point above, a plurality of regions including at least one matching point are identified to identify a target matching point for each region, and A cooking device that performs image registration between an image captured through the thermal imaging camera and an image captured through the optical camera based on the above target matching point.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the one or more processors, the cooking device, Identifying a matching score between feature points by comparing first information corresponding to each of the plurality of first feature points and second information corresponding to each of the plurality of second feature points, The first feature point and the second feature point, where the above-mentioned identified matching score exceeds a preset first score, are identified as matching points, and The above first information is, Information based on the difference in pixel values ​​between each of the above plurality of first feature points and adjacent pixels, and The above second information is, A cooking device, which is information based on the difference in pixel values ​​between each of the above-mentioned plurality of second feature points and adjacent pixels.

3. In Paragraph 2, The above matching point includes information regarding a first coordinate corresponding to a first feature point and a second coordinate corresponding to a second feature point, and When the above instructions are executed individually or collectively by the one or more processors, the cooking device, Identifying a mathematical model including rotation information and movement information for an image captured through an optical camera based on an image captured through a thermal imaging camera, based on the first coordinates and the second coordinates corresponding to the target matching point, and A cooking device that performs image registration between an image captured through the thermal imaging camera and an image captured through the optical camera based on the above mathematical model.

4. In Paragraph 3, When the above instructions are executed individually or collectively by the one or more processors, the cooking device, Among the above matching points, other matching points excluding the target matching point are input into the identified mathematical model to identify error information, and If the above error information is greater than or equal to a preset threshold value, the first feature point and the second feature point, where the identified matching score exceeds a preset second score, are identified as the matching points. A cooking device in which the second score is lower than the first score.

5. In Paragraph 4, When the above instructions are executed individually or collectively by the one or more processors, the cooking device, Among the above matching points, other matching points excluding the target matching point are input into the identified mathematical model to identify error information, and If the above error information is less than the above preset threshold value, the target matching point for each region among the above matching points is re-identified, and Re-identify the mathematical model based on the re-identified target matching point above, and A cooking device that identifies error information by inputting other matching points, excluding the re-identified target matching point among the above matching points, into the re-identified mathematical model.

6. In Paragraph 2, When the above instructions are executed individually or collectively by the one or more processors, the cooking device, Identify at least one fourth matching point located within a predetermined distance based on the third matching point among the above matching points, and A cooking device for identifying the matching point group including the third matching point and the fourth matching point.

7. In Paragraph 6, When the above instructions are executed individually or collectively by the one or more processors, the cooking device, The image captured by the thermal imaging camera is segmented into a plurality of regions each including the matching point group, and Based on the matching score between the above feature points and pixel response intensity, the above target matching point for each region is identified, and The image registration is performed based on the above target matching point, and A cooking device in which the pixel response intensity is the average value of the brightness difference for each of a plurality of pixels adjacent to the matching point.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the one or more processors, the cooking device, Among the plurality of pixels included in the image captured by the thermal imaging camera, a pixel in which the difference between the pixel values ​​corresponding to each adjacent pixel exceeds a preset range is identified as the first feature point, and A cooking device that identifies a pixel as a second feature point in which the difference between the pixel values ​​corresponding to each adjacent pixel among a plurality of pixels included in an image captured by the optical camera exceeds a preset range.

9. In Paragraph 1, When the above instructions are executed individually or collectively by the one or more processors, the cooking device, Identifying at least one of the position information and shooting direction of each of the thermal imaging camera and the optical camera, and A cooking device that acquires a second image converted based on the position information of each of the thermal imaging camera and the optical camera and the shooting direction of the thermal imaging camera, and the image acquired through the optical camera.

10. In Paragraph 1, When the above instructions are executed individually or collectively by the one or more processors, the cooking device, Based on the distance information between the second matching points above, at least one matching point group including at least one matching point is identified, and A cooking device that identifies a plurality of regions including each of the matching point groups in an image captured by the thermal imaging camera, and identifies a target matching point for each region.

11. In a method for controlling a cooking device, The operation of acquiring a first image group comprising a plurality of thermal images acquired through a thermal imaging camera and acquiring a second image group comprising a plurality of optical images acquired through an optical camera; An operation of identifying a plurality of first feature points in the first image group above; An operation of identifying a plurality of second feature points in the above second image group; An operation of identifying a plurality of first matching points based on a plurality of first feature points for a first thermal image among the first image group and a plurality of second feature points for a first optical image among the second image group; An operation of identifying a plurality of second matching points by adding a plurality of matching points identified based on a plurality of first feature points for a second thermal image among a first image group and a plurality of second feature points for a second optical image among a second image group to the plurality of first matching points; An operation of identifying a plurality of regions including at least one matching point based on the second matching point above, and identifying a target matching point for each region; and A control method comprising: an operation to perform image registration between an image captured through the thermal imaging camera and an image captured through the optical camera based on the above target matching point.

12. In Paragraph 11, The operation of identifying the above plurality of matching points is, An operation of identifying a matching score between feature points by comparing first information corresponding to each of the plurality of first feature points and second information corresponding to each of the plurality of second feature points; and The operation of identifying first feature points and second feature points as matching points, wherein the identified matching score exceeds a preset first score; further includes The above first information is, Information based on the difference in pixel values ​​between each of the above plurality of first feature points and adjacent pixels, and The above second information is, A control method based on information that is the difference in pixel values ​​between each of the above-mentioned plurality of second feature points and adjacent pixels.

13. In Paragraph 12, The above matching point includes information regarding a first coordinate corresponding to a first feature point and a second coordinate corresponding to a second feature point, and The operation of performing the above image registration is, An operation of identifying a mathematical model including rotation information and movement information for an image captured through an optical camera based on an image captured through a thermal imaging camera, based on the first coordinates and the second coordinates corresponding to the target matching point; and A control method further comprising: an operation to perform image registration between an image captured through the thermal imaging camera and an image captured through the optical camera based on the above mathematical model.

14. In Paragraph 13, An operation of identifying error information by inputting other matching points among the above matching points, excluding the target matching point, into the identified mathematical model; and If the above error information is greater than or equal to a preset threshold value, the operation of identifying the first feature point and the second feature point, in which the identified matching score exceeds a preset second score, as the matching point; is included. A control method in which the second score is a lower score than the first score.

15. A non-transient computer-readable storage medium storing computer instructions that cause said cooking device to perform an operation when executed by a processor of said cooking device, wherein said operation is, The operation of acquiring a first image group comprising a plurality of thermal images acquired through a thermal imaging camera and acquiring a second image group comprising a plurality of optical images acquired through an optical camera; An operation of identifying a plurality of first feature points in the first image group above; An operation of identifying a plurality of second feature points in the above second image group; An operation of identifying a plurality of first matching points based on a plurality of first feature points for a first thermal image among the first image group and a plurality of second feature points for a first optical image among the second image group; An operation of identifying a plurality of second matching points by adding a plurality of matching points identified based on a plurality of first feature points for a second thermal image among a first image group and a plurality of second feature points for a second optical image among a second image group to the plurality of first matching points; An operation of identifying a plurality of regions including at least one matching point based on the second matching point above, and identifying a target matching point for each region; and A non-transient computer-readable storage medium comprising: an operation to perform image registration between an image captured through the thermal imaging camera and an image captured through the optical camera based on the above target matching point.

Images