Image processing device and program

Abstract

According to one embodiment, an image processing device has: an input unit that receives input of point group data obtained by irradiation with irradiation light, an inference unit that infers, from the point group data, at least one of first point group data that is not dependent on the irradiation light and second point group data obtained by extracting only an irradiation light component, and a loss calculation unit that calculates the loss between the results of the inference by the inference unit and correct answer information for the point group data that was the subject of the inference by the inference unit and uses the results of the calculation to learn a parameter for a model used for inference by the inference unit.

Description

Image Processing Apparatus and Program

[0001] Embodiments of the present invention relate to an image processing apparatus and a program.

[0002] In recent years, various computer vision tasks (hereinafter referred to as CV tasks), such as image recognition or semantic segmentation, have been widely used. Generally, in these CV tasks, it is known that the performance of detection and estimation deteriorates due to solar shadows. In addition, it is also a problem that the detection performance may deteriorate due to sunlight conditions, such as the difference between the morning sun and the evening sun.

[0003] Therefore, techniques for intrinsic image decomposition, such as those disclosed in Non-Patent Document 1, which decompose an image into a component (albedo) that does not depend on the illumination light and a component (shade) that depends on the illumination light, have attracted attention. This intrinsic image decomposition is an ill-posed problem of decomposing an input image I into albedo (R) and shade (S) as shown in the following (1). Generally, in this intrinsic image decomposition, optimization is performed by imposing constraint conditions. I = R · S … (1)

[0004] In recent years, 3D data representation has been rapidly spreading, and among them, point cloud data has attracted attention from the viewpoints of its simplicity and geometry preservation. Even in the processing tasks of this point cloud data, since the accuracy may deteriorate depending on the sunlight conditions, an extension from intrinsic image decomposition to the processing of point cloud data is required. Hereinafter, the task related to this extension is called intrinsic point-cloud decomposition.

[0005] “Unsupervised Learning for Intrinsic Image Decomposition from a Single Image”<https: / / openaccess.thecvf.com / content_CVPR_2020 / papers / Liu_Unsupervised_Learning_for_Intrinsic_Image_Decomposition_From_a_Single_Image_CVPR_2020_paper.pdf>

[0006] On the other hand, since intrinsic image decomposition is intended solely for grid data processing, it cannot be directly extended to point clouds of non-grid data. Furthermore, applying conventional general point cloud processing models directly to point cloud decomposition tasks results in low inference performance because global solar information cannot be obtained.

[0007] This invention was made in view of the above circumstances, and its purpose is to provide an image processing device and program that can perform eigenpoint cloud decomposition with high accuracy.

[0008] An image processing apparatus according to one aspect of the present invention includes: an input unit that receives input of point cloud data irradiated by illumination light; an inference unit that infers from the point cloud data at least one of a first point cloud data independent of the illumination light and a second point cloud data in which only the component of the illumination light is extracted; and a loss calculation unit that calculates the loss between the result of the inference by the inference unit and the correct information of the point cloud data that is the target of the inference by the inference unit, and learns the parameters of the model used for inference by the inference unit using the result of this calculation.

[0009] According to the present invention, eigenpoint group decomposition can be performed with high accuracy.

[0010] Figure 1 is a diagram showing an example of the application of an image processing apparatus according to the first embodiment of the present invention. Figure 2 is a diagram showing an example of a model structure according to the first embodiment of the present invention. Figure 3 is a diagram showing an example of the application of an image processing apparatus according to the second embodiment of the present invention. Figure 4 is a diagram showing an example of a model structure according to the second embodiment of the present invention. Figure 5 is a diagram showing an example of the application of an image processing apparatus according to the third embodiment of the present invention. Figure 6 is a diagram showing an example of a model structure according to the third embodiment of the present invention. Figure 7 is a diagram illustrating the calculation of PLD. Figure 8 is a diagram showing an example of the application of an image processing apparatus according to the fourth embodiment of the present invention. Figure 9 is a diagram showing an example of a model structure according to the fourth embodiment of the present invention. Figure 10 is a block diagram showing an example of the hardware configuration of an image processing apparatus according to one embodiment of the present invention.

[0011] Embodiments relating to this invention will be described below. (First Embodiment) Figure 1 is a diagram showing an example of application of an image processing apparatus according to the first embodiment of the present invention. As shown in Figure 1, the image processing apparatus 100 according to the first embodiment of the present invention comprises a point cloud input unit 10, an image processing unit 20, and a loss calculation unit 30. The image processing unit 20 has a storage unit 21, an Albedo point cloud inference unit 22, and a Shadow point cloud inference unit 23.

[0012] Figure 2 shows an example of a model structure according to the first embodiment of the present invention. The point cloud input unit 10 receives a colored input point cloud I irradiated by any light source, including sunlight, and stores it in the storage unit 21.

[0013] The Albedo point cloud inference unit 22 infers an Albedo point cloud that is independent of the illumination light from the input point cloud I stored in the memory unit 21, using a model used for Albedo point cloud inference, and stores the inference result in the memory unit 21. The Albedo point cloud is represented as shown in (2) below.

[0014]

[0015] The Shadow point cloud inference unit 23 extracts only the illumination light component from the input point cloud I stored in the memory unit 21 to form a Shadow point cloud. Using a model used for Shadow point cloud inference, the inference unit 23 infers the result and stores it in the memory unit 21. The Shadow point cloud is represented as shown in (3) below.

[0016]

[0017] The Albedo point cloud inference unit 22 may infer the gray-scale Albedo point cloud first, and then the color Albedo point cloud. Similarly, the Shadow point cloud inference unit 23 may infer the gray-scale Shadow point cloud first, and then the color Shadow point cloud. Furthermore, both inference by the Albedo point cloud inference unit 22 and inference by the Shadow point cloud inference unit 23 may be performed based on the input point cloud I, or only one of them may be performed. The loss calculation unit 30 calculates the distance L_a, i.e., the loss, between Albedo A, which is the correct information for the Albedo point cloud, and the Albedo point cloud inference result stored in the storage unit 21, as defined in (4) below. The correct information for the Albedo point cloud is stored in advance in the storage unit 21, for example.

[0018]

[0019] The loss calculation unit 30 learns the parameters of the model used by the Albedo point cloud inference unit 22 for inferring the Albedo point cloud based on this distance. The loss calculation unit 30 also calculates the distance between Shadow S, which is the correct answer information for the Shadow point cloud, and the inference result for the Shadow point cloud stored in the memory unit 21, defining it as follows (5). The correct answer information for the Shadow point cloud is stored in the memory unit 21 beforehand, for example.

[0020]

[0021] The loss calculation unit 30 learns the parameters of the model used for inferring the Shadow point cloud by the Shadow point cloud inference unit 23 based on this distance. The generalized g(a, b) of the right-hand side of (4) or (5) above is a function of the error between a and b, and examples include the Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE).

[0022] As described above, by using the trained model and performing inference by the Albedo point cloud inference unit 22 or the Shadow point cloud inference unit 23, the point cloud can be accurately inferred, that is, accurately decomposed into eigenclouds. Furthermore, performance may be improved by using the loss calculation unit 30 to convert from the normal RGB color space to another color space such as Lab before calculating the loss mentioned above.

[0023] (Second Embodiment) Figure 3 is a diagram showing an example of application of the image processing apparatus according to the second embodiment of the present invention. As shown in Figure 3, the image processing apparatus 100 according to the second embodiment of the present invention includes a common feature extraction unit 24 in the image processing unit 20, in addition to the parts described in the first embodiment.

[0024] Figure 4 shows an example of a model structure according to a second embodiment of the present invention. In the first embodiment, the model used for inference by the Albedo point cloud inference unit 22 and the model used for inference by the Shadow point cloud inference unit 23 were implemented as independent models, and the Albedo point cloud and Shadow point cloud were inferred by separate processes. However, the Albedo point cloud and Shadow point cloud have common characteristics.

[0025] Therefore, in the second embodiment, the common feature extraction unit 24 extracts features common to the Albedo point cloud and the Shadow point cloud from the input point cloud I stored in the storage unit 21, and stores the extraction results in the storage unit 21.

[0026] Then, by using these extraction results to perform inference by the Albedo point cloud inference unit 22 and inference by the Shadow point cloud inference unit 23, the point cloud inference performance can be improved compared to the first embodiment. Also, similar to the first embodiment, the Albedo point cloud inference unit 22 may infer the gray-scale Albedo point cloud first, and then the color Albedo point cloud. Similarly, the Shadow point cloud inference unit 23 may infer the gray-scale Shadow point cloud first, and then the color Shadow point cloud.

[0027] (Third Embodiment) Figure 5 is a diagram showing an example of the application of the image processing apparatus according to the third embodiment of the present invention. As shown in Figure 5, the image processing apparatus 100 according to the third embodiment of the present invention includes a PLD calculation unit 25 in the image processing unit 20, in addition to the parts described in the second embodiment. Here, we will describe as an example a configuration in which the PLD calculation unit 25 is added to the configuration described in the second embodiment, that is, a configuration in which the PLD calculation unit 25 is added to the configuration that includes the common feature extraction unit 24 in the image processing unit 20, but a configuration in which the PLD calculation unit 25 is added to the configuration described in the first embodiment, that is, a configuration in which the PLD calculation unit 25 is added to the configuration that does not include the common feature extraction unit 24 in the image processing unit 20, is also acceptable.

[0028] Figure 6 shows an example of a model structure according to the third embodiment of the present invention. Figure 7 is a diagram illustrating the PLD calculation. In the model structures of the first and second embodiments described above, the distribution of light irradiating the entire point cloud is unknown, which can significantly degrade the point cloud inference performance.

[0029] Therefore, in the third embodiment, the Projected Light Distribution (PLD) for roughly determining the global light distribution, which is the distribution of irradiated light, is calculated as follows. The PLD calculation unit 25 projects the input point cloud I in an arbitrary direction D and obtains an image P_D based on the result of this projection. In an outdoor scene, if the input point cloud I is projected along the direction of sunlight, only the sunlit parts will be in the image P_D, resulting in a bright image P_D. Conversely, if the input point cloud I is projected in the direction opposite to the direction of sunlight, an image consisting only of shaded areas will be generated, resulting in a dark image P_D.

[0030] The PLD calculation unit 25 can roughly calculate the global light distribution by acquiring an image P_D for each of the multiple projection directions onto the input point cloud I and creating a distribution as a PLD.

[0031] The PLD calculation unit 25 stores the results of this calculation in the storage unit 21. The common feature extraction unit 24 uses the results of this calculation to extract features common to the Albedo point cloud and the Shadow point cloud, and stores the extraction results in the storage unit 21. The Albedo point cloud inference unit 22 estimates the Albedo point cloud using the extraction results from the common feature extraction unit 24 and the calculation results from the PLD calculation unit 25. The Shadow point cloud inference unit 23 estimates the Shadow point cloud using the extraction results from the common feature extraction unit 24 and the calculation results from the PLD calculation unit 25. As a result, the point cloud inference performance can be improved compared to the first and second embodiments.

[0032] (Fourth Embodiment) Figure 8 is a diagram showing an example of the application of the image processing apparatus according to the fourth embodiment of the present invention. As shown in Figure 8, the image processing apparatus 100 according to the fourth embodiment of the present invention includes a PLD processing unit 26 in the image processing unit 20, in addition to the parts described in the third embodiment. Here, a configuration in which a PLD calculation unit 25 and a PLD processing unit 26 are added to the configuration that includes the common feature extraction unit 24 described in the second embodiment is described as an example, but a configuration in which a PLD calculation unit 25 and a PLD processing unit 26 are added to a configuration that does not include the common feature extraction unit 24 is also acceptable.

[0033] Figure 9 shows an example of a model structure according to the fourth embodiment of the present invention. The PLD processing unit 26 processes the PLD calculated by the PLD calculation unit 25 and converts the PLD into a feature map as a feature quantity. Since the PLD is acquired while changing the projection direction, it is generally represented on a two-dimensional image or a hemisphere, but the PLD processing unit 26 processes using a processing model that matches the representation of the PLD. For example, when the PLD is represented on a hemisphere, Sphere Net, which processes hemispherical data with a virtual grid, is suitable, so the PLD processing unit 26 processes using this Sphere Net. Furthermore, by inputting the feature map obtained in this way into the Albedo point cloud inference unit 22, the Shadow point cloud inference unit 23, and the common feature extraction unit 24, the inference performance is improved compared to the third embodiment.

[0034] In other words, in the fourth embodiment, the common feature extraction unit 24 uses the processing results from the PLD processing unit 26 to extract features common to the Albedo point cloud and the Shadow point cloud, and stores the extraction results in the storage unit 21. The Albedo point cloud inference unit 22 uses the extraction results from the common feature extraction unit 24 and the processing results from the PLD processing unit 26 to estimate the Albedo point cloud. The Shadow point cloud inference unit 23 uses the extraction results from the common feature extraction unit 24 and the processing results from the PLD processing unit 26 to estimate the Shadow point cloud.

[0035] When an outdoor scene is assumed and PLD processing is added as in the fourth embodiment, the Shadow point cloud inference unit 23 may first obtain a gray-scale Shadow point cloud, then merge the features extracted from the PLD with the gray-scale Shadow point cloud to infer the color Shadow point cloud. Similarly, the Albedo point cloud inference unit 22 may first obtain a gray-scale Albedo point cloud, then merge the features extracted from the PLD with the gray-scale Albedo point cloud to infer the color Albedo point cloud. Performing the calculation in this manner may improve the accuracy of the inference.

[0036] Figure 10 is a block diagram showing an example of the hardware configuration of an image processing device according to one embodiment of the present invention. In the example shown in Figure 10, the image processing device 100 according to the above embodiment is configured by, for example, a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0037] The communication interface 114 includes, for example, one or more wireless communication interface units, enabling the transmission and reception of information with the communication network. As the wireless interface, for example, an interface employing a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0038] The input / output interface 113 is connected to an input device 200 and an output device 300, which are attached to the image processing device 100 and used by users or the like.

[0039] The input / output interface 113 captures operation data input by a user or the like through an input device 200 such as a keyboard, a touch panel, or a touchpad, and can perform processing to output the output data to an output device 300 including a display device using liquid crystal or organic EL (Electro Luminescence) or the like for display. Note that, for the input device 200 and the output device 300, a device built in the image processing apparatus 100 may be used, or an input device and an output device of another information terminal that can communicate with the image processing apparatus 100 via a network may be used.

[0040] The program memory 111B is used in combination with a non-volatile memory such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that can be written to and read from at any time, and a non-volatile memory such as a ROM (Read Only Memory) as a non-temporary tangible storage medium, and can store programs necessary for executing various control processes and the like according to an embodiment.

[0041] The data memory 112 is used in combination with a non-volatile memory such as the above and a volatile memory such as a RAM (Random Access Memory) as a tangible storage medium, and can be used to store various data or information acquired and created during various processes.

[0042] The image processing apparatus 100 according to an embodiment of the present invention can be configured as a data processing apparatus having processing function units by software, including each unit of the image processing apparatus 100, for example, the point group input unit 10, the image processing unit 20, and the loss calculation unit 30 shown in FIG.

[0043] The storage devices and storage unit 21 used as work memory for each part of the image processing device 100 may be configured using the data memory 112 shown in Figure 10. However, the storage area configured in these storage devices is not an essential component within the image processing device 100, and may be an area provided in an external storage medium such as a USB (Universal Serial Bus) memory, or in a storage device such as a database server located in the cloud.

[0044] All of the above processing functions can be implemented by having the hardware processor 111A read and execute a program stored in the program memory 111B. Some or all of these processing functions may be implemented in various other forms, including application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0045] Furthermore, the methods described in each embodiment can be stored as programs (software means) that can be executed by a computer on recording media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, MOs, etc.), and semiconductor memories (ROMs, RAMs, flash memories, etc.), and can also be transmitted and distributed via communication media. The programs stored on the media also include configuration programs that configure the computer to run software means (including not only the execution program but also tables or data structures). The computer implementing this device reads the program recorded on the recording media and, if necessary, constructs the software means using the configuration program, and executes the above-described processes by controlling the operation of this software means. Note that the recording media referred to in this specification are not limited to those for distribution, but also include storage media such as magnetic disks or semiconductor memories provided inside the computer or in devices connected via a network.

[0046] Note that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist thereof at the implementation stage. Also, the respective embodiments may be implemented in appropriate combination, and in that case, the combined effects can be obtained. Furthermore, the above embodiments include various inventions, and various inventions can be extracted by combinations selected from a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiments, if the problem can be solved and the effects can be obtained, the configuration from which these constituent elements are deleted can be extracted as an invention.

[0047] 100... Image processing apparatus 10... Point cloud input unit 20... Image processing unit 21... Storage unit 22... Albedo point cloud inference unit 23... Shadow point cloud inference unit 24... Common feature extraction unit 25... PLD calculation unit 26... PLD processing unit 30... Loss calculation unit

Claims

1. An image processing apparatus comprising: an input unit that receives input of point cloud data irradiated by illumination light; an inference unit that infers from the point cloud data at least one of a first point cloud data independent of the illumination light and a second point cloud data in which only the component of the illumination light is extracted; and a loss calculation unit that calculates the loss between the result of the inference by the inference unit and the correct information of the point cloud data that is the target of the inference by the inference unit, and learns the parameters of the model used for inference by the inference unit using the result of this calculation.

2. The image processing apparatus according to claim 1, further comprising a common feature extraction unit that extracts features common to the first point cloud data and the second point cloud data from the point cloud data for which the input has been received, wherein the inference unit infers at least one of the first point cloud data and the second point cloud data using the common features.

3. The image processing apparatus according to claim 1, further comprising a distribution calculation unit that calculates the distribution of the irradiated light by projecting the point cloud data received as input onto each of a plurality of directions, wherein the inference unit infers at least one of the first point cloud data and the second point cloud data using the distribution of the irradiated light.

4. The image processing apparatus according to claim 3, further comprising a conversion processing unit that performs a process of converting the distribution of the irradiated light into feature quantities, wherein the inference unit infers at least one of the first point cloud data and the second point cloud data using the feature quantities.

5. An image processing program that causes a processor to function as one of the components of the image processing apparatus according to any one of claims 1 to 4.

Images