Image processing device and image processing method

Abstract

The present invention determines the specific detection accuracy when detecting an object from image information image-captured by a vision sensor. 　An image processing device according to the present invention comprises: an image acquisition unit that acquires, from a vision sensor, image information which includes an object and was obtained by image-capturing the object using the vision sensor; an object detection unit that detects a region of the object from the image information; a circumscribed polygon calculation unit that calculates a circumscribed polygon for the detected region of the object; an area ratio calculation unit that calculates a first area of the detected region of the object and a second area of the circumscribed polygon, and calculates a first area ratio of the first area to the second area, or a second area ratio of the second area to the first area; and a detection accuracy determination unit that determines the detection accuracy of the object on the basis of the calculated first area ratio or second area ratio.

Description

Image processing apparatus and image processing method

[0001] This disclosure relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method for detecting an object from image information captured by a visual sensor.

[0002] Patent Document 1 describes an image recognition device that can speed up image recognition by pre-limiting the number of objects to be detected in detail, thereby reducing the amount of computation required compared to a case where the number of candidates is not limited. Specifically, Patent Document 1 describes an image recognition device that detects an object to be searched from an image containing the object to be searched, comprising: a segmentation means for separating and identifying objects independent of the image containing the object to be searched; a shape type attribute information calculation unit for extracting objects that approximate pre-classified typical shapes; a shape type attribute information bank means for storing the degree of approximation for each typical shape as a result of calculations by the shape type attribute information calculation unit; and a candidate object selection means for focusing on elements of a pre-determined typical shape for the object to be searched and determining a predetermined number of candidates as recognition candidates based on the stored values ​​of the shape type attribute information bank means.

[0003] Patent Document 2 describes a novel handling system with fewer drawbacks. Specifically, Patent Document 2 describes a robot system having a robot with a movable part and an end effector that grasps and moves an object, and a sensor that measures the object, and capable of processing the object in multiple operating modes. A data analysis unit analyzes the first data of the object obtained by measuring the object with the sensor and calculates the second data of the object. A database stores the second data of the object in association with the status of the object data, which is determined according to the operating mode of the robot system when the first data of the object was obtained. An operation instruction unit gives an operation instruction to the robot system that processes the object, in an operating mode determined according to the status of the object stored in the database, and according to the operation command determined according to the second data of the object.

[0004] Japanese Patent Publication No. Hei 8-315152 Japanese Patent Publication No. 2020-151780

[0005] When detecting a target object from image information captured by a visual sensor, if it is difficult to teach the shape of the target object one by one through methods such as pattern matching, for example, blob detection or detection based on artificial intelligence (AI) is used. However, it is difficult for users to confirm how accurate the detection results obtained by blob detection or AI-based detection are.

[0006] Therefore, when detecting a target object from image information captured by a visual sensor, there is a need for an image processing apparatus and an image processing method that can determine how accurate the detection accuracy is.

[0007] A representative first aspect of the present disclosure provides an image processing apparatus comprising: an image acquisition unit that acquires, from a visual sensor, image information containing a target object obtained by the visual sensor capturing an image of the target object; a target object detection unit that detects a region of the target object from the image information; a circumscribed polygon calculation unit that calculates a circumscribed polygon of the detected region of the target object; an area ratio calculation unit that calculates a first area of the detected region of the target object and a second area of the circumscribed polygon, and calculates a first area ratio of the first area to the second area, or a second area ratio of the second area to the first area; and a detection accuracy determination unit that determines the detection accuracy of the target object based on the calculated first area ratio or the calculated second area ratio.

[0008] A representative second aspect of the present disclosure provides an image processing method executed by a computer, the method comprising: a step of acquiring, from a visual sensor, image information containing a target object obtained by the visual sensor capturing an image of the target object; a step of detecting a region of the target object from the image information; a step of calculating a circumscribed polygon of the detected region of the target object; a step of calculating a first area of the detected region of the target object and a second area of the circumscribed polygon, and calculating a first area ratio of the first area to the second area, or a second area ratio of the second area to the first area; and a step of determining the detection accuracy of the target object based on the calculated first area ratio or the calculated second area ratio.

[0009] This is a block diagram showing an example configuration of an image processing apparatus according to the first embodiment of this disclosure. This is an explanatory diagram for explaining the process from imaging an object to detecting a circumscribed polygon in the image processing apparatus according to the first embodiment. This is a flowchart showing an example of an image processing method according to the first embodiment of this disclosure. This is a block diagram showing an example configuration of an image processing apparatus according to the second embodiment of this disclosure. This is a flowchart showing an example of an image processing method according to the second embodiment of this disclosure. This is a block diagram showing an example configuration of an image processing apparatus according to the third embodiment of this disclosure. This is a flowchart showing an example of an image processing method according to the third embodiment of this disclosure. This is a block diagram showing an example configuration of an image processing apparatus according to the fourth embodiment of this disclosure during learning. This is a block diagram showing an example configuration of an image processing apparatus according to the fourth embodiment of this disclosure after learning. This is a flowchart showing an example of an image processing method according to the fourth embodiment of this disclosure. This is a flowchart showing an example of three steps in the learning process in step S9 of the flowchart shown in Figure 10. This is a block diagram showing an example configuration of an image processing apparatus according to the fifth embodiment of this disclosure. This is a flowchart showing an example of an image processing method according to the fifth embodiment of this disclosure. This is a diagram showing the elliptical region of the detected object and the circumscribed polygon which is a circumscribed rectangle. This is a diagram showing a region where a part of the elliptical region of the detected object is missing and the circumscribed polygon CP which is a circumscribed rectangle.

[0010] Embodiments of the present disclosure will be described in detail below with reference to the drawings. (First Embodiment) Figure 1 is a block diagram showing an example configuration of an image processing apparatus according to the first embodiment of the present disclosure. The image processing apparatus 10 is connected to a visual sensor 20. The image processing apparatus 10 may also include a visual sensor 20. The visual sensor 20 is, for example, a camera that images an object. The visual sensor 20 is, for example, attached to a robot hand and images an object that the robot hand grasps, holds, or processes. The connection between the image processing apparatus 10 and the visual sensor 20 may be a wired connection or a wireless connection.

[0011] As shown in Figure 1, the image processing device 10 includes an image acquisition unit 101, an object detection unit 102, a circumscribed polygon calculation unit 103, an area ratio calculation unit 104, and a detection accuracy determination unit 105.

[0012] The image acquisition unit 101 acquires image information including the object obtained by the visual sensor 20 when the visual sensor captures the object, and outputs it to the object detection unit 102.

[0013] The object detection unit 102 detects the area of ​​the object from the image information and outputs the area of ​​the object to the circumscribed polygon calculation unit 103 and the area ratio calculation unit 104.

[0014] The circumscribed polygon calculation unit 103 calculates the circumscribed polygon of the area of ​​the object detected by the object detection unit 102, and outputs the calculated circumscribed polygon to the area ratio calculation unit 104. The circumscribed polygon can be, for example, a rectangle, square, pentagon, hexagon, etc.

[0015] The area ratio calculation unit 104 calculates the area of ​​the object region detected by the object detection unit 102 (which becomes the first area) and the area of ​​the circumscribed polygon calculated by the circumscribed polygon calculation unit 103 (which becomes the second area). It then calculates the ratio of the area of ​​the object region to the area of ​​the circumscribed polygon (which becomes the first area ratio), or the ratio of the area of ​​the circumscribed polygon to the area of ​​the object region (which becomes the second area ratio), and outputs the first area ratio or the second area ratio to the detection accuracy determination unit 105.

[0016] The detection accuracy determination unit 105 determines the detection accuracy of the object based on the first or second area ratio calculated by the area ratio calculation unit 104. The result of the detection accuracy determination is stored in the storage unit of the detection accuracy determination unit 105.

[0017] The operation of the image processing device 10 will be explained below using Figures 1 and 2, based on a specific example. Figure 2 is an explanatory diagram illustrating the process from imaging of an object to detecting its circumscribed polygon.

[0018] The visual sensor 20 captures an image of a rectangular object T. Figure 2 shows the top surface of the object T captured by the visual sensor 2, and the shape of the top surface of the object T is rectangular. The image acquisition unit 101 acquires image information including the object obtained by the visual sensor 20 capturing the object, and outputs it to the object detection unit 102.

[0019] The object detection unit 102 detects the region of an object from image information. Figure 2 shows the detected region P of the object. The shape of the region P of the object is a deformed rectangle, which is the shape of the top surface of the object T. This deformation from the rectangle is caused by, for example, at least one of the imaging conditions of the visual sensor 20 and the detection conditions of the object detection unit 102. For object detection, for example, blob detection or detection by artificial intelligence (AI) can be used.

[0020] Blob detection is described as "Blob Analysis" in, for example, "https: / / www.visco-tech.com / technical / direction-present / blob / ". "Blob Analysis" refers to a method of analyzing images that have undergone binarization, where grayscale images are converted to 0s and 1s based on an arbitrary threshold. For AI-based detection of regions such as masks (segmentations) from images, for example, Mask R-CNN, a multi-task learning model for object detection, is described in, for example, "https: / / ai-scholar.tech / articles / computer-vision / Mask-R-CNN".

[0021] The circumscribed polygon calculation unit 103 calculates the circumscribed polygon CP of the region P of the object detected by the object detection unit 102. Figure 2 shows an example where the circumscribed polygon CP is a circumscribed rectangle. When the circumscribed polygon CP is a circumscribed rectangle, the circumscribed rectangle is calculated as follows.

[0022] First, we find a convex polygon from a set of points. A method for finding a convex polygon from a set of points is the convex hull algorithm, which is an algorithm for finding the convex hull and is described at "https: / / ja.wikipedia.org / wiki / %E5%87%B8%E5%8C%85%E3%82%A2%E3%83%AB%E3%82%B4%E3%83%AA%E3%82%BA%E3%83%A0". To find the smallest circumscribing rectangle, we repeat the following steps (1) and (2) for each side of the found convex hull: (1) Draw a line perpendicular to the side and create a rectangle parallel to that line. (2) Calculate the area of ​​the rectangle and update the rectangle with the smallest area.

[0023] The area ratio calculation unit 104 calculates the area of ​​the object region P detected by the object detection unit 102 and the area of ​​the circumscribed polygon CP calculated by the circumscribed polygon calculation unit 103, and calculates the ratio of the area of ​​the object region to the area of ​​the circumscribed polygon (first area ratio) or the ratio of the area of ​​the circumscribed polygon to the area of ​​the object region (second area ratio). The following describes the case in which the area ratio calculation unit 104 calculates the first area ratio. For example, suppose the first area ratio of the area S1 of the object region P to the area S2 of the circumscribed polygon CP shown in Figure 2 is 85%.

[0024] The detection accuracy determination unit 105 determines that the detection accuracy is low if the first area ratio calculated by the area ratio calculation unit 104 is less than a certain value, and determines that the detection accuracy is high if the first area ratio is equal to or greater than a certain value. The certain value is a threshold for determining whether the detection accuracy is high or low. If the certain value set in the detection accuracy determination unit 105 is 90%, then when the first area ratio of the area S1 of the object region P to the area S2 is 85%, the detection accuracy is determined to be low.

[0025] A certain value for determining detection accuracy is set in advance; for example, the manufacturer sets this value before or at the time of shipment. If the area ratio is defined as the ratio of the area of ​​the circumscribed polygon to the area of ​​the object's region (second area ratio), then if the area ratio exceeds a certain value, it is determined that the detection accuracy is low, and if the area ratio is below a certain value, it is determined that the detection accuracy is high.

[0026] Next, an example of an image processing method according to the first embodiment of the present disclosure will be described. Figure 3 is a flowchart of an example of an image processing method according to the first embodiment of the present disclosure. In the following description, an example in which the image processing method of the present disclosure is performed by an image processing device 10 will be described, but the image processing method of the present disclosure can also be performed by a configuration other than the image processing device 10.

[0027] In step S1, the image acquisition unit 101 acquires image information including the object, obtained by the visual sensor 20 capturing an image of the object, from the visual sensor 20.

[0028] In step S2, the object detection unit 102 detects the region of the object from the image information.

[0029] In step S3, the circumscribed polygon calculation unit 103 calculates the circumscribed polygon of the region of the object detected by the object detection unit 102.

[0030] In step S4, the area ratio calculation unit 104 calculates the area of ​​the region of the object detected by the object detection unit 102 and the area of ​​the circumscribed polygon calculated by the circumscribed polygon calculation unit 103.

[0031] Subsequently, in step S5, the area ratio calculation unit 104 calculates the ratio of the area of ​​the object's region to the area of ​​the circumscribed polygon (first area ratio). The area ratio calculation unit 104 may also calculate the ratio of the area of ​​the circumscribed polygon to the area of ​​the object's region (second area ratio).

[0032] In step S6, the detection accuracy determination unit 105 determines the detection accuracy of the object based on the area ratio calculated by the area ratio calculation unit 104, and terminates the process.

[0033] According to the image processing apparatus and image processing method of this embodiment described above, it is possible to determine how accurate the detection is when detecting an object from image information captured by a visual sensor.

[0034] (Second Embodiment) Figure 4 is a block diagram showing an example configuration of an image processing apparatus according to the second embodiment of the present disclosure. As shown in Figure 4, the configuration of the image processing apparatus 11 in this embodiment is the same as the configuration of the image processing apparatus 10 shown in Figure 1, with the addition of a threshold setting unit 106. In this embodiment, the threshold setting unit 106 accepts an operation by the user to specify a threshold and sets the threshold in the detection accuracy determination unit 105. The threshold setting unit 106 accepts a first threshold for determining whether the first area ratio is above a certain value, or a second threshold for determining whether the second area ratio is below a certain value, and sets the first threshold or the second threshold in the detection accuracy determination unit 105. The operation of the image processing apparatus 11 in this embodiment is the same as the operation of the image processing apparatus 10, except for the operation of the threshold setting unit 106. The user can input a threshold to the threshold setting unit 106, for example, by operating a keyboard or by operating the setting screen of a liquid crystal display with a touch panel.

[0035] Figure 5 is a flowchart showing an example of an image processing method according to the second embodiment of the present disclosure. In this embodiment, as shown in Figure 5, a step S0 is added before step S1 in the flowchart shown in Figure 3, in which the threshold setting unit 106 determines whether or not a threshold has been set in the detection accuracy determination unit 105.

[0036] In step S0, the threshold setting unit 106 determines whether or not a threshold has been set in the detection accuracy determination unit 105 based on user operation. If a threshold has been set, the process proceeds to step S1; otherwise, the process ends. Even if the threshold setting unit 106 has not set a threshold in the detection accuracy determination unit 105 based on user operation, the process may proceed to step S1 if an initial value for the threshold has been set in the detection accuracy determination unit 105.

[0037] In addition to the effects of the first embodiment, the image processing apparatus and image processing method of this embodiment described above allow the user to set a threshold value. For example, the user can change the setting value arbitrarily depending on the shape of the object, etc.

[0038] (Third Embodiment) Figure 6 is a block diagram showing an example configuration of an image processing apparatus according to the third embodiment of the present disclosure. As shown in Figure 6, the configuration of the image processing apparatus 12 in this embodiment is the same as that of the image processing apparatus 10 shown in Figure 1, with the addition of a detection accuracy determination result output unit 107. In this embodiment, the detection accuracy determination result output unit 107 outputs the detection accuracy determination result of the detection accuracy determination unit 105 to the outside. The operation of the image processing apparatus 12 in this embodiment is the same as the operation of the image processing apparatus 10, except for the operation of the detection accuracy determination result output unit 107.

[0039] The detection accuracy determination result output unit 107 outputs the detection accuracy determination result to an external device, such as a liquid crystal display unit or a printer unit, or to an external device via a network. When outputting to an external device via a network, the detection accuracy determination result output unit 107 becomes a communication unit.

[0040] Figure 7 is a flowchart showing an example of an image processing method according to the third embodiment of the present disclosure. In this embodiment, as shown in Figure 7, a step S7 is added after step S6 of the flowchart shown in Figure 3, in which the detection accuracy determination result output unit 107 outputs the detection accuracy determination result of the detection accuracy determination unit 105 to the outside.

[0041] According to the image processing apparatus and image processing method of this embodiment described above, in addition to the effects of the first embodiment, the user can know the detection accuracy determination result.

[0042] (Fourth Embodiment) This embodiment is an example in which the object detection unit detects an object using deep learning. Figure 8 is a block diagram showing an example configuration of the image processing apparatus of the fourth embodiment of this disclosure during the learning phase. Figure 9 is a block diagram showing an example configuration of the image processing apparatus of the fourth embodiment of this disclosure after learning.

[0043] As shown in FIGS. 8 and 9, the configuration of the image processing apparatus 13 of the present embodiment is obtained by adding a circumscribed polygon teaching unit 108, a learning unit 109, and a learning model updating unit 110 to the configuration of the image processing apparatus 10 shown in FIG. 1. The circumscribed polygon teaching unit 108, the learning unit 109, and the learning model updating unit 110 operate during learning for updating a learning model, and do not operate in normal operation after learning. In FIG. 9, paths that sequentially connect the detection accuracy determination unit 105, the circumscribed polygon teaching unit 108, the learning unit 109, the learning model updating unit 110, and the object detection unit 102 do not function after learning, so these paths are represented by broken lines. Further, in FIG. 9, paths connecting between the image acquisition unit 101 and the learning unit 109 and between the circumscribed polygon calculation unit 103 and the circumscribed polygon teaching unit 108 do not function after learning, so these paths are represented by broken lines.

[0044] In order to disable the operation of the circumscribed polygon teaching unit 108, the learning unit 109, and the learning model updating unit 110, for example, the signal output side of each path shown by a broken line may be configured not to output a signal, the signal input side may be configured not to receive a signal, or a switch may be provided in each path to switch between conduction and interruption of a signal. In order to prevent learning for updating the learning model from being performed, it is not necessary to block all of the paths. For example, if the path from the learning model updating unit 110 to the object detection unit 102 is blocked, the learning model will not be updated.

[0045] The learning targets an object for which the outer shape of the object included in the image information is equal or similar to the circumscribed polygon calculated by the circumscribed polygon calculation unit 103. An example of an object whose outer shape is equal or similar to the circumscribed polygon is a rectangular parallelepiped. When the object is a rectangular parallelepiped, if the outer shape of the object included in the image information acquired by the object detection unit 102 is a rectangle, and the circumscribed polygon calculated by the circumscribed polygon calculation unit 103 is also a rectangle, then the outer shape of the object and the circumscribed polygon are equal. When the object is a rectangular parallelepiped, depending on imaging conditions such as the exposure conditions of the visual sensor 20 or the image size, the outer shape of the object included in the image information acquired by the object detection unit 102 becomes a deformed rectangle. If the circumscribed polygon calculated by the circumscribed polygon calculation unit 103 is a rectangle, then the outer shape of the object and the circumscribed polygon are similar. In the following description, it is assumed that the outer shape of the object included in the image information is a rectangle, and the circumscribed polygon is also a rectangle.

[0046] In the present embodiment, during learning, the object detection unit 102 uses deep learning to detect an object. Initially, the object detection unit 102 detects a region of the object from image information using a learning model preset by a manufacturer or a user, and outputs the region of the object (whose outer shape is a rectangle) to the circumscribed polygon calculation unit 103 and the area ratio calculation unit 104. The circumscribed polygon calculation unit 103 outputs a rectangle as the circumscribed polygon to the area ratio calculation unit 104 and the circumscribed polygon teaching unit 108.

[0047] As described in the first embodiment, the area ratio calculation unit 104 outputs a first area ratio or a second area ratio to the detection accuracy determination unit 105. The detection accuracy determination unit 105 determines the detection accuracy of the object based on the first area ratio or the second area ratio calculated by the area ratio calculation unit 104, and outputs the determination result to the circumscribed polygon teaching unit 108.

[0048] When the detection accuracy determination unit 105 determines that the detection accuracy is low, the circumscribed polygon teaching unit 108 outputs the rectangular circumscribed polygon detected by the circumscribed polygon calculation unit 103 to the learning unit 109 as teaching data.

[0049] The learning unit 109 stores the image information output from the image acquisition unit 101, and uses the circumscribing polygon of the object's region as teaching data for the image information containing the object that the detection accuracy determination unit 105 has determined to have low detection accuracy, and learns a learning model.

[0050] The learning model update unit 110 updates the learning model of the object detection unit 102, using the model learned by the learning unit 109 as the learning model used by the object detection unit 102. By updating the learning model, the learning model update unit 110 ensures that the object detection result changes even when the same image is detected the next time.

[0051] The operation of the circumscribed polygon teaching unit 108, the learning unit 109, and the learning model update unit 110 continues until the detection accuracy determination unit 105 determines that the detection accuracy is high, and a learning model that increases detection accuracy is set in the object detection unit 102. Once a learning model that increases detection accuracy is set in the object detection unit 102, the image processing device 13 terminates the learning operation and returns to normal operation. In normal operation, the setting that targeted objects whose outline in the image information is equal to the circumscribed polygon calculated by the circumscribed polygon calculation unit 103 is canceled during learning.

[0052] Figure 10 is a flowchart showing an example of an image processing method according to the fourth embodiment of this disclosure. Figure 11 is a flowchart showing an example of three steps of the learning process in step S9 of the flowchart shown in Figure 10.

[0053] As shown in Figure 10, the image processing method of this embodiment adds steps S8 and S9 after step S5 in the flowchart shown in Figure 3, instead of step S6.

[0054] In step S8, the detection accuracy determination unit 105 determines the detection accuracy of the object based on the area ratio calculated by the area ratio calculation unit 104. If it determines that the detection accuracy is low, it proceeds to step S9. If it determines that the detection accuracy is high, it terminates the process.

[0055] Step S9 is a learning step in which the object detection unit 102 learns a learning model, and as shown in Figure 11, step S9 comprises steps S91, S92, and S93.

[0056] In step S91, if the detection accuracy determination unit 105 determines that the detection accuracy is low, the circumscribed polygon teaching unit 108 outputs the circumscribed polygon of the rectangle detected by the circumscribed polygon calculation unit 103 to the learning unit 109 as teaching data.

[0057] In step S92, the learning unit 109 stores the image information output from the image acquisition unit 101, and uses the circumscribing polygon of the object's region as teaching data for the image information containing the object that the detection accuracy determination unit 105 has determined to have low detection accuracy, and learns a learning model.

[0058] In step S93, the learning model update unit 110 updates the learning model of the object detection unit 102 to use the model learned by the learning unit 109 as the learning model for use in the object detection unit 102, and the image processing device 13 moves on to step S2.

[0059] According to the image processing apparatus and image processing method of this embodiment described above, in addition to the effects of the first embodiment, the learning model of the object detection unit can be updated based on the results of the detection accuracy determination unit.

[0060] (Fifth Embodiment) Figure 12 is a block diagram showing an example configuration of an image processing apparatus according to the fifth embodiment of the present disclosure. As shown in Figure 12, the configuration of the image processing apparatus 14 in this embodiment is the same as the configuration of the image processing apparatus 10 shown in Figure 1, with the addition of a condition changing unit 111. In this embodiment, the condition changing unit 111 changes at least one of the imaging conditions of the visual sensor 20 and the detection conditions of the object detection unit 102 when the detection accuracy determination unit 105 determines that the detection accuracy is low.

[0061] The operation of the image processing device 14 in this embodiment is the same as the operation of the image processing device 10, except for the operation of the condition change unit 111. The imaging conditions of the visual sensor 20 can be changed, for example, by changing the exposure conditions, image size, etc. of the visual sensor 20. The detection conditions of the object detection unit 102 can be changed, for example, by changing the setting of the binarization threshold in blob detection or the conditions for detection by AI.

[0062] Figure 13 is a flowchart showing an example of an image processing method according to the fifth embodiment of this disclosure. As shown in Figure 13, the image processing method of this embodiment includes an additional step S10, which performs processing if step S9 in the flowchart shown in Figure 10 is "YES" in step S8.

[0063] In step S8, the detection accuracy determination unit 105 determines the detection accuracy of the object based on the area ratio calculated by the area ratio calculation unit 104. If it determines that the detection accuracy is low, it proceeds to step S10. If it determines that the detection accuracy is high, it terminates the process.

[0064] In step S10, if the detection accuracy determination unit 105 determines that the detection accuracy is low, the condition change unit 111 changes at least one of the imaging conditions of the visual sensor 20 and the detection conditions of the object detection unit 102, and then proceeds to step S1.

[0065] According to the image processing apparatus and image processing method of this embodiment described above, in addition to the effects of the first embodiment, if the detection accuracy determination unit determines that the detection accuracy is low, it is possible to automatically perform a process to change at least one of the imaging conditions and detection conditions.

[0066] The first to fifth embodiments described above can be combined as appropriate. For example, the threshold setting unit 106 of the second embodiment may be added to the image processing devices 12, 13, and 14 of the third to fifth embodiments, and the detection accuracy determination output unit 107 of the third embodiment may be added to the image processing devices 11, 13, and 14 of the second, fourth, and fifth embodiments. Also, the circumscribed polygon teaching unit 108, learning unit 109, and learning model update unit 110 of the fourth embodiment may be added to the image processing devices 11, 12, and 14 of the second, third, and fifth embodiments, and the condition change unit 111 of the fifth embodiment may be added to the image processing devices 11, 12, and 13 of the second to fourth embodiments.

[0067] In order to realize the functional blocks included in the image processing apparatus in each embodiment described above, the image processing apparatus can be implemented by hardware, software, or a combination thereof. Similarly, the image processing method can also be implemented by hardware, software, or a combination thereof. Here, implementation by software means implementation by a computer reading and executing a program.

[0068] To implement the components included in an image processing device through software or a combination thereof, the image processing device includes an arithmetic processing unit such as a CPU (Central Processing Unit). The arithmetic processing unit functions as an execution unit. The image processing device also includes an image processing storage device such as an HDD (Hard Disk Drive) that stores application software or various control programs such as an OS (Operating System), and a main memory such as RAM (Random Access Memory) for storing data temporarily required for the arithmetic processing unit to execute the program.

[0069] The image processing device then reads application software or an operating system from the image processing memory, expands the read application software or OS into the main memory, and performs calculations based on this application software or OS. Furthermore, it controls various hardware components of the image processing device based on these calculation results. This realizes the functional blocks of this embodiment. The image processing method can also be realized with a configuration similar to that of the image processing device.

[0070] The components included in an image processing device can be realized by hardware, including electronic circuits. When an image processing device is configured as hardware, some or all of the functions of each component included in the image processing device can be implemented using integrated circuits (ICs) such as ASICs (Application Specific Integrated Circuits), gate arrays, FPGAs (Field Programmable Gate Arrays), and CPLDs (Complex Programmable Logic Devices).

[0071] Programs can be stored and supplied to a computer using various types of non-transitor computer-readable media. Non-transitor computer-readable media include various types of tangible storage media. Examples of non-transitor computer-readable media include magnetic recording media (e.g., hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memory)). Furthermore, the program may be supplied to the computer by various types of temporary computer-readable media.

[0072] According to the image processing apparatus and image processing method of this disclosure, including the embodiments described above, it is possible to determine how accurate the detection is when detecting an object from image information captured by a visual sensor.

[0073] While the embodiments described above are preferred embodiments of the present invention, the scope of the present invention is not limited to these embodiments, and various modifications can be made to implement the invention without departing from the spirit of the invention.

[0074] For example, in the first to fifth embodiments, the object was a rectangular parallelepiped, but the shape of the object is not particularly limited, and may be a flat plate, a cube, a cylinder, or an elliptical prism. The operation of the circumscribed polygon calculation unit 103, the area ratio calculation unit 104, and the detection accuracy determination unit 105 when the object is an elliptical prism and the shape of the upper surface of the object is elliptical will be explained with reference to Figures 14 and 15.

[0075] Figure 14 shows the elliptical region P1 of the detected object and the circumscribed rectangle CP. Figure 15 shows region P2, where a part of the elliptical region of the object is missing (region S), and the circumscribed rectangle CP.

[0076] The circumscribed polygon calculation unit 103 calculates the circumscribed polygon CP of the elliptical region P1 of the detected object. The circumscribed polygon CP is a bounding rectangle. The area ratio calculation unit 104 calculates the area of ​​the object region P1 and the area of ​​the circumscribed polygon CP, and calculates the area ratio (first area ratio) of the area of ​​the object region to the area of ​​the circumscribed polygon. In the example shown in Figure 14, the calculated area ratio is 80%, and in the example shown in Figure 15, the calculated area ratio is 75%. If the threshold for determining whether the detection accuracy is high or low is 78%, the detection accuracy determination unit 105 determines that the detection accuracy is high in the example shown in Figure 14, and low in the example shown in Figure 15. Here, the detection accuracy was determined by calculating the area ratio of the object's region to the area of ​​the circumscribed polygon (first area ratio), but the detection accuracy could also be determined by calculating the area ratio of the circumscribed polygon's region to the area of ​​the object's region (second area ratio).

[0077] With respect to the above embodiment, the following additional information is disclosed. (Addendum 1) An image processing apparatus comprising: an image acquisition unit (101) that acquires image information including an object obtained by the visual sensor (20) imaging the object from the visual sensor; an object detection unit (102) that detects the region of the object from the image information; a circumscribed polygon calculation unit (103) that calculates the circumscribed polygon of the detected region of the object; an area ratio calculation unit (104) that calculates a first area of ​​the detected region of the object and a second area of ​​the circumscribed polygon, and calculates a first area ratio of the first area to the second area, or a second area ratio of the second area to the first area; and a detection accuracy determination unit (105) that determines the detection accuracy of the object based on the calculated first area ratio or second area ratio.

[0078] (Note 2) The image processing apparatus (11) according to Note 1, further comprising a threshold setting unit (106) that receives a first threshold for determining whether the first area ratio is greater than or equal to a certain value, or a second threshold for determining whether the second area ratio is less than or equal to a certain value, and sets the first threshold or the second threshold in the detection accuracy determination unit (105).

[0079] (Note 3) The image processing apparatus according to Note 1 or 2, further comprising a detection accuracy determination result output unit (107) that outputs the determination result from the detection accuracy determination unit.

[0080] (Note 4) The image processing apparatus according to Note 1, wherein the object detection unit (102) detects an object using deep learning, and the object detection unit (102) determines that the detection accuracy is low, the image processing apparatus comprises: an circumscribed polygon teaching unit (108) that takes the circumscribed polygon of the object's region as teaching data for image information showing an object whose outline is equal to or similar to the circumscribed polygon calculated by the circumscribed polygon calculation unit; and a learning unit (109) that learns the teaching data taught by the circumscribed polygon teaching unit.

[0081] (Note 5) The image processing apparatus according to any one of Notes 1 to 4, further comprising a condition changing unit (111) that changes at least one of the imaging conditions of the visual sensor (20) and the detection conditions of the object detection unit (102) when the detection accuracy determination unit (105) determines that the detection accuracy is low.

[0082] (Note 6) An image processing method in which a computer performs the following steps: acquiring image information including an object from a visual sensor (20) obtained by the visual sensor capturing an image of the object; detecting the region of the object from the image information; calculating the circumscribing polygon of the detected region of the object; calculating a first area of ​​the detected region of the object and a second area of ​​the circumscribing polygon, and calculating a first area ratio of the first area to the second area, or a second area ratio of the second area to the first area; and determining the detection accuracy of the object based on the calculated first area ratio or second area ratio.

[0083] 10, 11, 12, 13, 14 Image processing device 20 Visual sensor 101 Image acquisition unit 102 Object detection unit 103 Circumscribed polygon calculation unit 104 Area ratio calculation unit 105 Detection accuracy determination unit 106 Threshold setting unit 107 Detection accuracy determination result output unit 108 Circumscribed polygon teaching unit 109 Learning unit 110 Learning model update unit 111 Condition change unit

Claims

1. An image processing apparatus comprising: an image acquisition unit that acquires image information including an object obtained by the visual sensor capturing an object from the visual sensor; an object detection unit that detects the region of the object from the image information; a circumscribed polygon calculation unit that calculates the circumscribed polygon of the detected region of the object; an area ratio calculation unit that calculates a first area of ​​the detected region of the object and a second area of ​​the circumscribed polygon, and calculates a first area ratio of the first area to the second area, or a second area ratio of the second area to the first area; and a detection accuracy determination unit that determines the detection accuracy of the object based on the calculated first area ratio or second area ratio.

2. The image processing apparatus according to claim 1, further comprising a threshold setting unit that receives a first threshold for determining whether the first area ratio is greater than or equal to a certain value, or a second threshold for determining whether the second area ratio is less than or equal to a certain value, and sets the first threshold or the second threshold in the detection accuracy determination unit.

3. The image processing apparatus according to claim 1 or 2, further comprising a detection accuracy determination result output unit that outputs the determination result obtained by the detection accuracy determination unit.

4. The image processing apparatus according to claim 1, wherein the object detection unit detects an object using deep learning, and the object detection unit detects an object for which the outline of the object included in the image information is equal to or similar to the circumscribed polygon calculated by the circumscribed polygon calculation unit, the apparatus comprises: a circumscribed polygon teaching unit that takes the circumscribed polygon of the object's region as teaching data for image information of an object for which the detection accuracy determination unit has determined to have low detection accuracy; and a learning unit that learns the teaching data taught by the circumscribed polygon teaching unit.

5. The image processing apparatus according to any one of claims 1 to 4, further comprising a condition changing unit that changes at least one of the imaging conditions of the visual sensor and the detection conditions of the object detection unit when the detection accuracy determination unit determines that the detection accuracy is low.

6. An image processing method comprising: a computer performing the steps of: acquiring image information including an object from a visual sensor obtained by the visual sensor capturing an image of the object; detecting the region of the object from the image information; calculating the circumscribing polygon of the detected region of the object; calculating a first area of ​​the detected region of the object and a second area of ​​the circumscribing polygon, and calculating a first area ratio of the first area to the second area, or a second area ratio of the second area to the first area; and determining the detection accuracy of the object based on the calculated first area ratio or second area ratio.

Images