Image processing device and program

Abstract

An image processing device according to one embodiment of the present invention comprises: an acquisition unit that acquires a captured image captured by an imaging device; a separation unit that separates, from the captured image, a reflection layer composed of a reflection component reflected by a transparent body; a detection unit that uses the reflection layer to detect a rectangular region and an imaging source region that includes reflection of an imaging source; and an assessment unit that, on the basis of the positional relationship between the rectangular region and the imaging source region, assesses whether the rectangular region is a transparent body region.

Description

Image processing apparatus, and program

[0001] One aspect of the present invention relates to an image processing apparatus and a program.

[0002] In order for vehicles equipped with autonomous driving, industrial robots, care robots, drones, etc. to move safely in the human living space, it is necessary to accurately identify transparent areas such as glass doors and glass windows.

[0003] As a practical technology, it is possible to detect a transparent area as a mere obstacle using a thermal sensor, an ultrasonic sensor, or the like. However, in order to determine that the detected area is a glass door or a glass window, it is necessary to combine the detection result by the sensor with the camera image, and further, a process of performing object recognition is required for the determination of the transparent area. Mounting a plurality of sensors and cameras on the device increases the size, weight, and cost of the device.

[0004] As a technology for detecting a transparent area, for example, there is Non-Patent Document 3. This technology attempts to create learning data from a large amount of transparent data sets and detect the transparent area by a CNN-based deep learning architecture. Since most of the transparent data sets in this technology are data of plastic bottles, cups, etc. photographed from a distance of about 80 cm, it is possible to detect plastic bottles and cups of the same size to some extent, but it is not suitable for detecting glass doors and glass windows. At present, there is no decisive factor for the image-based transparent area detection technology.

[0005] Qiming Hu et al., “Single Image Reflection Separation via Component Synergy”, [online], [Retrieved January 23, 2025], Internet <https: / / openaccess.thecvf.com / content / ICCV2023 / papers / Hu_Single_Image_Reflection_Separation_via_Component_Synergy_ICCV_2023_paper.pdf> Tianhe Ren et al., “Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks,” [online], [Retrieved January 23, 2025], Internet.<https: / / arxiv.org / pdf / 2401.14159> Shreeyak S. Sajjan et al., “ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation”, [online], [Retrieved January 23, 2025], Internet<https: / / arxiv.org / pdf / 1910.02550>

[0006] This invention was made in view of the above circumstances, and its objective is to provide a technique for determining transparent regions while simplifying the apparatus.

[0007] An image processing apparatus according to one aspect of the present invention comprises: an acquisition unit that acquires a captured image taken by a shooting device; a separation unit that separates a reflective layer consisting of a reflective component reflected by a transparent object from the captured image; a detection unit that uses the reflective layer to detect a rectangular region and a source region including reflections of the source of shooting; and a determination unit that determines whether or not the rectangular region is a transparent object region based on the positional relationship between the rectangular region and the source region of shooting.

[0008] According to one aspect of the present invention, a technique for determining the transparent region can be provided while simplifying the apparatus.

[0009] Figure 1 is a block diagram of an image processing apparatus according to the first embodiment of the present invention. Figure 2 is a block diagram showing an example of the hardware configuration of the image processing apparatus. Figure 3 is a flowchart illustrating the operation of the image processing apparatus according to the first embodiment of the present invention. Figure 4 is a diagram showing an example of a captured image. Figure 5 is a diagram showing an example of a reflective layer. Figure 6 is a diagram showing an example of a transparent layer. Figure 7 is a diagram showing an example of a residue layer. Figure 8 is a diagram showing an example of object detection by the object detection unit. Figure 9 is a diagram illustrating an example of the determination logic of the transparent area determination unit. Figure 10 is a diagram illustrating an example of the determination process by the transparent area determination unit. Figure 11 is a flowchart illustrating the operation of the transparent area determination unit. Figure 12 is a flowchart illustrating the operation of the transparent area determination unit. Figure 13 is a block diagram of an image processing apparatus according to the second embodiment of the present invention. Figure 14 is a flowchart illustrating the operation of the image processing apparatus according to the second embodiment of the present invention. Figure 15 is a flowchart illustrating the operation of the image processing apparatus according to the third embodiment of the present invention.

[0010] Embodiments will be described below with reference to the drawings. The embodiments shown below are illustrative examples of devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention is not defined by the shape, structure, arrangement, etc. of the components. Each functional block can be realized as hardware, software, or a combination of both. It is not essential that each functional block is distinguished as in the following example. For example, some functions may be performed by functional blocks other than the illustrative functional blocks. Furthermore, the illustrative functional blocks may be further divided into finer functional subblocks. In the following description, elements having the same function and configuration are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] [1] First Embodiment [1-1] Diagram 1 of the configuration of the image processing apparatus 1 is a block diagram of the image processing apparatus 1 according to the first embodiment of the present invention. The image processing apparatus 1 comprises a shooting device 10, a control unit 20, a storage unit 30, an input unit 40, and an output unit 50.

[0012] The imaging device 10 consists of a camera and the like. The imaging device 10 captures (also called image capture) an image based on user operation. The imaging device 10 outputs the captured image as a captured image. In this embodiment, the imaging device 10 captures an image so that it includes a transparent object in the background. The transparent object includes glass doors and glass windows. The imaging device 10 also captures an image so that the photographer and / or the imaging equipment (including the imaging device 10) are reflected in the image. Reflection is a phenomenon in which, due to the reflection of light caused by a transparent object such as glass, another unintended image component is superimposed as a reflected component on the original image (background component). In this specification, the photographer and / or the imaging equipment are referred to as the source of the image. The captured image captured by the imaging device 10 includes the transparent object and the source of the image reflected in this transparent object.

[0013] The control unit 20 comprises an image acquisition unit 21, a reflective layer separation unit 22, an object detection unit 23, and a transparent object region determination unit 24. The image acquisition unit 21 is also referred to as the acquisition unit. The reflective layer separation unit 22 is also referred to as the separation unit. The object detection unit 23 is also referred to as the detection unit. The transparent object region determination unit 24 is also referred to as the determination unit.

[0014] The image acquisition unit 21 acquires images from the imaging device 10. The image acquisition unit 21 stores the acquired images in the storage unit 30. The control unit 20 can perform image processing using the images stored in the storage unit 30.

[0015] The reflective layer separation unit 22 separates the reflective layer of the captured image using the captured image acquired by the image acquisition unit 21. The reflective layer separation unit 22 performs predetermined processing using the first machine learning model 31.

[0016] The object detection unit 23 uses the reflective layer separated by the reflective layer separation unit 22 to detect predetermined objects and regions within the reflective layer. The object detection unit 23 performs predetermined processing using the second machine learning model 32.

[0017] The transparent object region determination unit 24 determines the transparent object region using the region detected by the object detection unit 23.

[0018] The storage unit 30 stores various data necessary for the control unit 20 to process. The storage unit 30 stores the first machine learning model 31 and the second machine learning model 32.

[0019] The first machine learning model 31 is a pre-trained model. The first machine learning model 31 may be a deep learning model. The first machine learning model 31 is trained to be able to separate the reflective layer from the captured image.

[0020] The second machine learning model 32 is a pre-trained model. The second machine learning model 32 may also be a deep learning model. The second machine learning model 32 is trained to detect and segment any object specified in the text.

[0021] The input unit 40 receives information entered by the user. The input unit 40 includes, for example, a keyboard and a mouse.

[0022] The output unit 50 outputs the data generated by the control unit 20. The output unit 50 includes, for example, a liquid crystal display device or an organic EL (Electro Luminescence) display device.

[0023] (Hardware configuration of the image processing device 1) Figure 2 is a block diagram showing an example of the hardware configuration of the image processing device 1.

[0024] The image processing device 1 can be configured using a computer. The image processing device 1 includes a processor 60, a program storage unit 61, a data storage unit 62, a communication interface unit (communication I / F unit) 63, an input / output interface unit (input / output I / F unit) 64, an input unit 40, and an output unit 50. The data storage unit 62, the program storage unit 61, the communication interface unit 63, and the input / output interface unit 64 are connected to the processor 60 via a bus 65.

[0025] The processor 60 consists of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and the like. The processor 60 executes the operations of the control unit 20 described above.

[0026] The program storage unit 61 includes, for example, a non-volatile memory that can be written to and read at any time, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and a non-volatile memory such as ROM (Read Only Memory). The program storage unit 61 stores the programs necessary for the processor 60 to perform various processes according to this embodiment. That is, the image acquisition unit 21, the reflective layer separation unit 22, the object detection unit 23, and the transparent object region determination unit 24 described above are realized by having the processor 60 execute the programs stored in the program storage unit 61.

[0027] The data storage unit 62 includes, for example, non-volatile memory such as an HDD or SSD, and volatile memory such as RAM (Random Access Memory). The data storage unit 62 temporarily stores various data acquired and created during the process of the processor 60 executing various processes. The data storage unit 62 is also used as a workspace for the processor 60.

[0028] The communication interface unit 63 includes a wired communication module and / or a wireless communication module. The wireless communication module includes a wireless LAN (Local Area Network). The communication interface unit 63 performs interface processing with an external device according to a predetermined communication standard. The communication interface unit 63 is capable of receiving information from an external device and transmitting information to an external device.

[0029] The input / output interface unit 64 is connected to the input unit 40 and the output unit 50. The input / output interface unit 64 performs interface processing with the input unit 40 and the output unit 50 according to a predetermined communication standard.

[0030] The captured images taken by the aforementioned imaging device 10 are received by the communication interface unit 63 or the input / output interface unit 64.

[0031] [1-2] Operation Next, the operation of the image processing device 1 configured as described above will be explained. Figure 3 is a flowchart illustrating the operation of the image processing device 1.

[0032] The imaging device 10 captures an image (captured image) that includes a transparent object based on the user's operation.

[0033] The image acquisition unit 21 acquires captured images (specifically, image data) from the imaging device 10 (step S100). The image acquisition unit 21 stores the acquired captured images in the storage unit 30. The control unit 20 can read the captured images stored in the storage unit 30 and perform various processing.

[0034] Next, the reflective layer separation unit 22 separates the reflective layer from the captured image (step S101). For example, the technique for separating the reflective layer from the captured image can be found in Non-Patent Document 1. The reflective layer separation unit 22 performs the process in step S101 using the first machine learning model 31.

[0035] The reflective layer separation unit 22 separates the captured image into a reflective layer, a transmission layer, and a residue layer. The reflective layer is the image reflected by the transparent object and projected onto the transparent object. The transmission layer is the background image of the transparent object (the image visible through the transparent object). The residue layer is the remaining layer other than the detected reflective layer and transmission layer. The image of the reflective object can be represented as the sum of the background component and the reflective component. The first machine learning model 31 is generated using the feature quantities of the reflective layer, transmission layer, and residue layer, respectively.

[0036] Figure 4 shows an example of a captured image. Figure 5 shows an example of a reflective layer. Figure 6 shows an example of a transparent layer. Figure 7 shows an example of a residue layer.

[0037] In the image in Figure 4, the background is visible through the glass door, and the photographer is reflected in the glass door. In the reflective layer in Figure 5, the photographer reflected in the glass door is separated from the image. In the transparent layer in Figure 6, the photographer reflected in the glass is removed from the image. In the residue layer in Figure 7, objects other than the reflective and transparent layers are separated.

[0038] Next, the object detection unit 23 uses the reflective layer separated in step S101 to detect the rectangular region and the source region (step S102). For example, the technique for detecting a specific object from an image can be found in Non-Patent Document 2. The object detection unit 23 uses the second machine learning model 32 to perform the processing in step S102.

[0039] The rectangular region is a region with a rectangular shape contained within the reflective layer. The rectangular region is not limited to a perfect rectangle and includes errors such as straight lines. The source region includes the reflection component of the source. The detection accuracy by the object detection unit 23 is set according to the second machine learning model 32. The object detection unit 23 may further detect objects that are thought to exist only in the reflective component of a transparent object as the source.

[0040] The objects to be detected by the object detection unit 23 can be arbitrarily set by the user. The objects to be detected are set in advance by the user via text input. Information about the objects to be detected is stored in the storage unit 30. The object detection unit 23 detects the objects specified by the user via text within the reflective layer and segments the detected objects. The object detection unit 23 identifies the segmented objects with rectangular frames. The second machine learning model 32 is generated using the feature quantities of the rectangular objects and the feature quantities of the source of the image (photographer and / or photographic equipment).

[0041] Figure 8 shows an example of object detection by the object detection unit 23. In Figure 8, object detection is performed using the reflective layer shown in Figure 5. In Figure 8, the source area and rectangular areas 1 to 3 are each divided by rectangular frames. In this embodiment, the source area is detected as the photographer (person). Rectangular areas 1 and 2 are reflective components other than the person. Rectangular area 3 is a glass door. From Figure 8, the photographer and the glass door (rectangular area 3 in Figure 8) can be detected.

[0042] In step S102, the rectangular region and the source region may be detected using the residue layer separated in step S101. Alternatively, the rectangular region and the source region may be detected using an image obtained by mixing the information of the reflective layer and the residue layer. For example, a method of adding up the brightness on a pixel-by-pixel basis can be used to mix the reflective layer and the residue layer.

[0043] Subsequently, the transparent object area determination unit 24 determines the transparent object area based on the positional relationship between the rectangular area detected in step S102 and the shooting source area (step S103). Specifically, when the rectangular area and the shooting source area overlap, the transparent object area determination unit 24 determines the rectangular area as the transparent object area. The transparent object area is the area occupied by a transparent object (such as a glass door or a glass window) in the captured image. On the other hand, when the rectangular area and the shooting source area do not overlap, the transparent object area determination unit 24 determines that the rectangular area is not a transparent object area (or it is unknown whether it is a transparent object area).

[0044] FIG. 9 is a diagram for explaining an example of the determination logic of the transparent object area determination unit 24. "rect1" represents the coordinates of the detected rectangular area, and "rect2" represents the coordinates of the detected shooting source area. By using the determination logic in FIG. 9, it is possible to determine whether the rectangular area and the shooting source area overlap or not.

[0045] FIG. 10 is a diagram for explaining an example of the determination process by the transparent object area determination unit 24. In FIG. 10, an example of the rectangular area and the shooting source area is shown. In FIG. 10, the transparent object area determination unit 24 determines that the rectangular area and the shooting source area overlap. Then, the transparent object area determination unit 24 determines the rectangular area as the transparent object area.

[0046] Subsequently, the output unit 50 outputs the determination result by the transparent object area determination unit 24 based on the control of the control unit 20 (step S104).

[0047] [1-3] Modified Example Next, a modified example for improving the determination accuracy of the rectangular area and the shooting source area will be described.

[0048] When detecting the shooting source area, the object detection unit 23 generates score information of the shooting source area. Also, when detecting the rectangular area, the object detection unit 23 generates score information of the rectangular area. The score information is a numerical value (score) related to the reliability indicating how accurate the result of object detection is, that is, how much the shape looks like the actual shape. Each score information has a higher score as the reliability increases, for example, with a maximum of 1. These score information are sent to the transparent object area determination unit 24.

[0049] The transparent object region determination unit 24 determines the accuracy of the source region and rectangular region detected by the object detection unit 23. Figures 11 and 12 are flowcharts illustrating the operation of the transparent object region determination unit 24. The processes in Figures 11 and 12 are included in step S103 of Figure 3.

[0050] As shown in Figure 11, the transparent area determination unit 24 determines the accuracy of the source area detected by the object detection unit 23. The transparent area determination unit 24 determines whether the score of the source area is equal to or greater than a first threshold (step S200). The first threshold can be set arbitrarily. If the score of the source area is equal to or greater than the first threshold (step S200 = Yes), the transparent area determination unit 24 determines whether the area of ​​the source area is smaller than the area of ​​the rectangular area (step S201). If the area of ​​the source area is smaller than the area of ​​the rectangular area (step S201 = Yes), the transparent area determination unit 24 determines that the source area detected by the object detection unit 23 includes the source image (step S202).

[0051] On the other hand, if the score of the source region is lower than the first threshold (step S200 = No), the transparent region determination unit 24 determines that the source region detected by the object detection unit 23 does not contain the source image (step S203). Also, if the area of ​​the source region is greater than or equal to the area of ​​the rectangular region (step S201 = No), the transparent region determination unit 24 determines that the source region detected by the object detection unit 23 does not contain the source image (step S203). In these cases, the transparent region determination unit 24 terminates the determination process for the source region to be determined.

[0052] Note that step S200 and step S201 may be performed to determine only one of them.

[0053] As shown in Figure 12, the transparent area determination unit 24 determines the accuracy of the rectangular area detected by the object detection unit 23. The transparent area determination unit 24 determines whether the score of the rectangular area is equal to or greater than the second threshold (step S300). The second threshold can be set arbitrarily. If the score of the rectangular area is equal to or greater than the second threshold (step S300 = Yes), the transparent area determination unit 24 determines whether the area of ​​the rectangular area is equal to or greater than the third threshold (step S301). The third threshold can be set arbitrarily. If the area of ​​the rectangular area is equal to or greater than the third threshold (step S301 = Yes), the transparent area determination unit 24 continues the process of determining the positional relationship between the rectangular area and the source area (step S302).

[0054] On the other hand, if the score of the rectangular region is lower than the second threshold (step S300 = No), the transparent region determination unit 24 excludes the rectangular region from the determination target (step S303). Also, if the area of ​​the rectangular region is smaller than the third threshold (step S301 = No), the transparent region determination unit 24 excludes the rectangular region from the determination target (step S303). Then, the transparent region determination unit 24 terminates the determination process.

[0055] If there are multiple rectangular regions, steps S300 to S303 are performed for each rectangular region. Note that either step S300 or step S301 may be used to determine only one of them.

[0056] [1-4] Effects of the First Embodiment According to the first embodiment, it is possible to determine the area of ​​transparent objects (such as glass doors and glass windows) present in the background of a camera using information from a single camera (shooting device 10). This makes it possible to simplify the device and reduce its size, weight, and cost.

[0057] Furthermore, the reflective layer containing the reflective component is separated from the image captured by the imaging device 10. Then, the rectangular region and the source region are detected using the reflective layer. This improves the accuracy of detecting transparent objects. In addition, it improves the accuracy of determining the transparent object region.

[0058] Furthermore, the accuracy of the source region is determined based on the area of ​​the rectangular region detected from the reflective layer and the source region. This improves the accuracy of determining the transparent region.

[0059] Furthermore, the system determines whether or not to exclude a rectangular region from the analysis based on its area. This improves the accuracy of detecting transparent regions.

[0060] [2] Second Embodiment In the second embodiment, the object detection process is performed on the captured image without separating the reflective layer from the captured image.

[0061] Figure 13 is a block diagram of an image processing apparatus 1 according to a second embodiment of the present invention. The control unit 20 includes an image acquisition unit 21, an object detection unit 23, and a transparent object region determination unit 24.

[0062] The object detection unit 23 uses the captured image acquired by the image acquisition unit 21 to detect a rectangular region and the source region. The object detection unit 23 performs predetermined processing using the second machine learning model 32.

[0063] The transparent object region determination unit 24 determines the transparent object region using the region detected by the object detection unit 23.

[0064] The memory unit 30 includes a second machine learning model 32. The second machine learning model 32 is trained to detect and segment any object specified in text.

[0065] Figure 14 is a flowchart illustrating the operation of the image processing device 1. The operation in step S400 in Figure 14 is the same as in step S100 in Figure 3.

[0066] Next, the object detection unit 23 uses the captured image acquired in step S400 to detect the rectangular region and the source region (step S401). The object detection unit 23 performs object detection using the second machine learning model 32.

[0067] Next, the transparent area determination unit 24 determines the transparent area based on the positional relationship between the rectangular area detected in step S401 and the source area (step S402). Specifically, the transparent area determination unit 24 determines the rectangular area to be a transparent area if the rectangular area and the source area overlap.

[0068] Next, the output unit 50 outputs the determination result from the transparent area determination unit 24 based on the control of the control unit 20 (step S403).

[0069] According to the second embodiment, transparent regions included in the captured image can be determined with less processing compared to the first embodiment.

[0070] Furthermore, the process for determining the area of ​​the rectangular region and the source region described in the first embodiment may be applied to the second embodiment.

[0071] [3] Third Embodiment The third embodiment detects a rectangular region in the captured image and detects the source region in the reflective layer. Then, the transparent region is determined based on the positional relationship between the rectangular region and the source region.

[0072] The block diagram of the image processing apparatus 1 according to the third embodiment is the same as in Figure 1. The object detection unit 23 detects a rectangular region using the captured image acquired by the image acquisition unit 21. The object detection unit 23 also detects the source region using the reflective layer separated by the reflective layer separation unit 22. The object detection unit 23 performs predetermined processing using the second machine learning model 32.

[0073] Figure 15 is a flowchart illustrating the operation of the image processing device 1. The operations in steps S500 and S501 in Figure 15 are the same as those in steps S100 and S101 in Figure 3.

[0074] Next, the object detection unit 23 detects a rectangular region using the captured image acquired in step S500 (step S502). The object detection unit 23 performs object detection using the second machine learning model 32.

[0075] Next, the object detection unit 23 detects the source region using the reflective layer separated in step S501 (step S503). The object detection unit 23 performs object detection using the second machine learning model 32.

[0076] Next, the transparent area determination unit 24 determines the transparent area based on the positional relationship between the rectangular area detected in step S502 and the source area detected in step S503 (step S504). Specifically, the transparent area determination unit 24 determines the rectangular area to be a transparent area if the rectangular area and the source area overlap.

[0077] Next, the output unit 50 outputs the determination result from the transparent area determination unit 24 based on the control of the control unit 20 (step S505).

[0078] According to the third embodiment, the same effects as the first embodiment can be obtained. The determination process described in the modified example of the first embodiment may be appropriately applied to the third embodiment.

[0079] Each process according to the above-described embodiment can be stored as a program (software means) that can be executed by a computer, for example, on a storage medium such as a magnetic disk, optical disk, or semiconductor memory, or transmitted and distributed via a communication medium. The storage medium includes a storage medium provided in a computer or a storage medium provided in a device connected via a network. The computer can then read the program stored in the storage medium and execute the above-described process by having its operation controlled by the read program.

[0080] The present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention.

[0081] 1…Image processing device 10…Photography device 20…Control unit 21…Image acquisition unit 22…Reflection layer separation unit 23…Object detection unit 24…Transparent object region determination unit 30…Storage unit 31…Machine learning model 32…Machine learning model 40…Input unit 50…Output unit 60…Processor 61…Program storage unit 62…Data storage unit 63…Communication interface unit 64…Input / output interface unit 65…Bus

Claims

1. An image processing apparatus comprising: an acquisition unit that acquires an image captured by a shooting device; a separation unit that separates a reflective layer consisting of a reflective component reflected by a transparent object from the captured image; a detection unit that uses the reflective layer to detect a rectangular region and a source region including reflections of the source; and a determination unit that determines whether or not the rectangular region is a transparent object region based on the positional relationship between the rectangular region and the source region.

2. The image processing apparatus according to claim 1, wherein the determination unit determines that the rectangular region is a transparent region when the rectangular region and the source region overlap.

3. The image processing apparatus according to claim 1, wherein the determination unit determines that the source region includes the source image when the area of ​​the source region is smaller than the area of ​​the rectangular region.

4. A program for causing a computer to function as each part of the image processing apparatus according to any one of claims 1 to 3.

Images