Image processing system, image processing method, and image processing program

Abstract

This image processing system (1) comprises an acquisition unit (50) and a learning unit (60). The acquisition unit (50) acquires image data of a plurality of compressed images corresponding to variations in the relative position of an optical element having a fine structure with respect to an imaging element in the stage subsequent to the optical element. The learning unit (60) uses the image data of the plurality of compressed images acquired by the acquisition unit (50) to generate a reconstruction processing model (30) for reconstructing the image data of a hyperspectral image from the image data of an image obtained by the imaging element.

Description

Image processing system, image processing method, and image processing program

[0001] The present invention relates to an image processing system, an image processing method, and an image processing program.

[0002] In recent years, research on compressive sensing type hyperspectral cameras has been underway (see Patent Documents 1 and 2). For example, a compressive sensing type hyperspectral camera performs optical encoding (operation / arithmetic of optical information) on an imaging target, captures an image, and uses reconstruction processing utilizing the sparsity of natural images to restore spatial information and wavelength information. In addition, techniques for improving the accuracy of reconstruction processing in a compressive sensing type hyperspectral camera have been provided (see Patent Document 3).

[0003] International Publication No. 2022 / 162801, International Publication No. 2022 / 162800, International Publication No. 2021 / 234797

[0004] However, there is room for improvement in the prior art. For example, in the above prior art, in a compressive sensing type hyperspectral camera, when the relative position of an optical element with respect to an image sensor changes, the accuracy of reconstruction processing decreases. For example, the accuracy of reconstruction processing decreases due to an attachment error of an optical element having a fine structure or an attachment error of an image sensor.

[0005] The present invention has been made in view of the above, and an object thereof is to suppress a decrease in the accuracy of reconstruction processing of a hyperspectral image in a compressive sensing type hyperspectral camera.

[0006] In order to solve the above-described problems and achieve the object, an image processing system includes an acquisition unit and a learning unit. The acquisition unit acquires image data of a plurality of compressed images corresponding to fluctuations in the relative position of an optical element with respect to an image sensor located downstream of an optical element having a fine structure. The learning unit generates a reconstruction processing model that reconstructs image data of a hyperspectral image from image data of an image obtained by the image sensor using the image data of the plurality of compressed images acquired by the acquisition unit.

[0007] According to the present invention, it is possible to suppress the decrease in accuracy of the hyperspectral image reconstruction process in a compressed sensing type hyperspectral camera.

[0008] Figure 1 is a diagram showing an example of the configuration of an image processing system according to the embodiment. Figure 2 is a diagram showing an example of the configuration of an imaging device that uses a reconstruction processing model generated in the learning process in the image processing system according to the embodiment. Figure 3 is a diagram for explaining the mounting error of the optical element to the housing according to the embodiment. Figure 4 is a diagram for explaining the overview of the learning process according to the embodiment. Figure 5 is a flowchart of the learning process procedure according to the embodiment. Figure 6 is a diagram showing another example of the configuration of the image processing system according to the embodiment. Figure 7 is a diagram showing an example of a computer that executes an information processing program in the image processing system according to the embodiment.

[0009] Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited by this embodiment. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0010] [Embodiment] [Image Processing System] The outline of the image processing system according to the embodiment will be described below. Figure 1 is a diagram showing an example of the configuration of the image processing system according to the embodiment.

[0011] As shown in Figure 1, the image processing system 1 according to this embodiment includes an image data memory 10, a camera image generation simulator 20, a reconstruction processing model 30, an error calculator 40, an acquisition unit 50, and a learning unit 60. Note that the configuration of the image processing system 1 shown in Figure 1 is merely an example, and the image processing system 1 can be configured in any way that allows it to perform the desired processing.

[0012] The image processing system 1 performs training on the reconstruction processing model 30 used in a compressed sensing type hyperspectral camera. Below, an imaging device, which is an example of a compressed sensing type hyperspectral camera in which the reconstruction processing model 30 is used, will be described, and then the image processing system 1 will be described in detail.

[0013] [Imaging Device] Figure 2 is a diagram showing an example of the configuration of an imaging device in which the reconstruction processing model 30 generated in the learning process in the image processing system 1 according to the embodiment is used. As shown in Figure 2, the imaging device 200 includes an imaging unit 210, an image processing unit 220, an image data memory 230, and a display 240.

[0014] [Imaging Unit 210] The imaging unit 210 has the function of an imaging unit that captures images. The imaging unit 210 has an image sensor 211, an optical element 212, and a housing 213. In the imaging unit 210, the image sensor 211 and the optical element 212 are mounted on the housing 213.

[0015] The image sensor 211 has a photoelectric conversion element such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The optical element 212 is an optical component having a microstructure. For example, the optical element 212 is a metasurface (metalens), but details will be described later. Although not shown in Figure 2, the imaging device 200 has a signal processing unit that processes the photoelectric conversion signal output from the image sensor 211 to generate an image signal (corresponding to the compressed image CI in Figure 2).

[0016] As shown in Figure 2, in the imaging unit 210, light such as natural light or illumination light is shone onto the object to be imaged (actual image), and the light transmitted / reflected / scattered by the object to be imaged 2, or the light emitted from the object to be imaged 2, forms an optical image on the image sensor 211 by the optical element 212.

[0017] As described above, the optical element 212 having a microstructure comprises a transparent substrate 215 and a plurality of structures 214. In Figure 2, one example shows the plurality of structures 214 arranged on the transparent substrate 215 in the planar direction of the transparent substrate 215, but the plurality of structures 214 may also be arranged within the transparent substrate 215 in the planar direction of the transparent substrate 215.

[0018] The optical element 212 has a microstructure. For example, the optical element 212 consists of a fine binary structure. In the example shown in Figure 2, the optical element 212 has a plurality of fine columnar structures 214. With this configuration, the optical element 212 has different response characteristics depending on the wavelength.

[0019] The optical element 212 performs optical encoding by having different imaging characteristics depending on the wavelength. Encoding, for example, is the conversion or modulation of an original signal into a different signal. The optical element 212 is a lens (wavelength-dependent PSF lens) with a PSF (Point Spread Function) of distinctly different shapes depending on the wavelength, and has the function of generating an image by applying different convolution operations to the real image (subject) for each wavelength. For example, if the optical element 212 is a wavelength-dependent PSF lens and an object is imaged with this optical element 212, a convolution operation is performed on the real image with different PSFs for each wavelength, and the result is imaged on the image sensor 211. In this way, the image sensor 211 acquires an observed image in which different convolution operations have been performed for each wavelength by the optical element 212, which is a wavelength-dependent PSF lens.

[0020] For example, the image sensor 211 has multiple pixels, each containing a photoelectric conversion element, arranged in a two-dimensional array. An example of a photoelectric conversion element is a photodiode (PD). Each pixel corresponds to red (R), green (G), and blue (B). An example of the wavelength band for red light is the wavelength λ. ０ Therefore, 600 nm < λ ０ ≤ 700 nm. An example of the wavelength range for green light is 500 nm < λ ０ ≤ 600 nm. An example of the wavelength range for blue light is 400 nm ≤ λ ０ The wavelength is ≤500 nm. Pixels R, G, and B may be in a Bayer array. Alternatively, the pixels may be for monochrome images.

[0021] The incident light reaches the image sensor 211 via the optical element 212. The charge generated at each pixel of the image sensor 211 is converted into an electrical signal that forms the basis of the pixel signal by a transistor (not shown) or the like. Note that the process of converting light into an electrical signal and generating digital data, which is image data (also simply called "image"), is a known technique, so a detailed explanation is omitted.

[0022] The optical element 212 is positioned on the side into which light from the imaging target 2 is incident. For example, the optical element 212 is provided so as to cover the image sensor 211 when viewed from above (front view). The optical element 212 is composed of a plurality of structures 214 on the bottom surface of the transparent substrate 215, for example, periodically (having a periodic structure). The transparent substrate 215 is, for example, SiO ２ This is a low refractive index transparent substrate made of materials such as (refractive index n = 1.45).

[0023] For example, the optical element 212 is composed of a plurality of microstructures (corresponding to the structure 214) having a width less than or equal to the wavelength of light when viewed from above (for example, a width smaller than the wavelength of light to be received by the image sensor 11). For example, the optical element 212 (or its structure 214) has a rotationally symmetric shape (for example, a four-fold rotationally symmetric shape) when viewed from above (in cross-section). The optical element 212 can employ any structure as long as it has the desired characteristics, and may have a two-dimensional structure or a three-dimensional structure. The optical element 212 can control the phase and light intensity according to the characteristics of light (wavelength, polarization, angle of incidence) simply by changing the parameters of this structure 214.

[0024] The optical element 212 has different PSFs depending on the wavelength, and its imaging characteristics (degree of blurring) differ depending on the wavelength of light from the imaging target 2. The light from the imaging target 2 is imaged onto the image sensor 211 by the optical element 212 with wavelength-dependent PSF function and acquired as an image (RGB image or monochrome image).

[0025] The image captured by the imaging unit 210, which is equipped with the optical element 212 described above (compressed image CI, etc.), corresponds to the result of optical convolution calculation performed on the imaging target 2 (real image) for each wavelength by the wavelength-dependent PSF of the optical element 212, and integration along the wavelength dimension on the pixel. The optical element 212 and the image sensor 211 capture (acquire) an image in an optically encoded and compressed state. In the case of the image sensor 211 being a color image sensor, after the optical convolution calculation described above, multiplication is performed according to the wavelength sensitivity of each R, G, and B pixel of the image sensor 211, and then integration along the wavelength dimension is performed on the pixel. In this way, the imaging unit 210 forms an optically encoded image on the image sensor 211 using a single optical element 212. In other words, the imaging unit 210 can perform effective encoding in spectral image reconstruction using a single optical element 212.

[0026] The imaging unit 210, specifically the image sensor 211, converts the incident light into, for example, 256 × 256 × 3 image data (electrical signals) and outputs it. In the notation of the image data, x × y represents x pixels × y pixels, and x × y × z (for example, x, y, and wavelength in Figure 2) represents x pixels × y pixels × z bands (channels). For example, if z is 3, it becomes 3-band color image data. For example, when capturing a monochrome image (z = 1), the image sensor 211 converts the incident light into, for example, 256 × 256 × 1 image data (electrical signals) and outputs it.

[0027] For example, in the imaging unit 210, the distance between the optical element 212 and the image sensor 211 is determined by the focal length of the lens, just like in a normal imaging device. Therefore, the size of the imaging unit 210 is equivalent to that of a normal camera with the same field of view F-number. The imaging unit 210 may be equipped with known components such as an infrared light cut optical filter, an electronic shutter, a viewfinder, a power supply (battery), and a flashlight, but their explanation is omitted as it is not particularly necessary for understanding the present invention. Furthermore, the above configuration is merely an example, and in the embodiment, known elements can be appropriately combined and used as components other than the optical element 212 and the image sensor 211. For example, commercially available HS (hyperspectral) cameras measure HS information (hyperspectral image) using a spectrometer (diffraction grating, etc.) and an optical filter. In contrast, an imaging device with a metasurface measures (estimates) HS information by capturing a compressed image and restoring it through reconstruction processing.

[0028] [Image Processing Unit 220] The image processing unit 220 has a reconstruction processing model 30. The reconstruction processing model 30 is a model (machine learning model) that takes compressed image CI, which is an image captured by the image sensor 211, as input and outputs image data of an output image, which is a hyperspectral image HI, which has a larger number of bands (channels) than the compressed image CI, by performing a reconstruction process on the input image data. The image processing unit 220 generates hyperspectral image HI image data by performing an inference process using the reconstruction processing model 30.

[0029] The reconstruction processing model 30 generates hyperspectral image data with tens of bands or more by performing a reconstruction process on the input image data. The reconstruction processing model 30 generates hyperspectral image data with 20 to 50 bands. In the following explanation, we will describe the case where the number of bands in the hyperspectral image is 31, and image data of a 256 × 256 × 31 hyperspectral image is generated as an example, but the number of bands in the hyperspectral image is not limited to 31 bands, but can be any number of bands, and may be tens or hundreds of bands.

[0030] For example, the reconstruction processing model 30 may be a neural network (NN) model (network) such as a deep neural network (DNN). For example, the reconstruction processing model 30 may have a structure related to a convolutional neural network (CNN). For example, the reconstruction processing model 30 may have a structure related to transposed convolution (deconvolution).

[0031] The above is merely an example, and the network structure (internal structure) of the reconstruction processing model 30 can be any structure as long as it can convert the input image into the desired hyperspectral image. For example, the reconstruction processing model 30 may be a neural network corresponding to the mathematical model of reconstruction processing described below.

[0032] [Image data memory 230] The image data memory 230 functions as a storage unit that stores images inferred (generated) by the inference process. For example, the image data memory 230 stores hyperspectral image data of, for example, 256 × 256 × 31, i.e., 31-band hyperspectral image data, generated by the reconstruction processing model 30.

[0033] [Display 240] The display 240 functions as a display unit that displays images inferred (generated) by the inference process. The display 240 displays hyperspectral images corresponding to hyperspectral image data provided via the image data memory 230.

[0034] [Image Processing System 1] As described above, the image processing system 1 shown in Figure 1 comprises an image data memory 10, a camera image generation simulator 20, a reconstruction processing model 30, an error calculator 40, an acquisition unit 50, and a learning unit 60.

[0035] The image processing system 1 performs a learning process for the reconstruction processing model 30 to suppress a decrease in the accuracy of the reconstruction process due to fluctuations in the relative position of the optical element 212 with respect to the image sensor 211. Fluctuations in the relative position of the optical element 212 with respect to the image sensor 211 are caused, for example, by mounting errors of the optical element 212 to the housing 213 or mounting errors of the image sensor 211 to the housing 213.

[0036] Mounting errors of the optical element 212 to the housing 213 can occur due to, for example, rotational misalignment, center misalignment, tilt, and Z-direction misalignment. Rotational misalignment is the deviation from the reference position in the rotational direction with the center of the optical element 212 as the center of rotation. Center misalignment is the deviation of the center of the optical element 212 from the reference position. Tilt is the tilt of the optical element 212 in the Z-direction from the reference position. Z-direction misalignment is the deviation of the optical element 212 in the Z-direction from the reference position. As shown in Figure 2, the Z-direction is the direction in which the image sensor 211 and the optical element 212 face each other, and is the direction in which light is incident on the imaging unit 210. Similarly, mounting errors of the image sensor 211 to the housing 213 can also occur due to, for example, rotational misalignment, center misalignment, tilt, and Z-direction misalignment.

[0037] Figure 3 is a diagram illustrating the mounting error of the optical element 212 to the housing 213 according to the embodiment. Figure 3(a) shows the standard compression process (PSF) assumed during learning, and Figures 3(b) to (d) show the variation in the compression process (PSF) caused by the mounting error of the optical element 212 to the housing 213. As shown in Figures 3(b) to (d), when a mounting error of the optical element 212 to the housing 213 occurs, a change in the compression process (PSF) occurs from the state shown in Figure 3(a).

[0038] In the following, the image data memory 10, camera image generation simulator 20, reconstruction processing model 30, error calculator 40, acquisition unit 50, and learning unit 60 will be described in detail in that order.

[0039] [Image Data Memory 10] The image data memory 10 has a function as a storage unit that stores the image data of a plurality of images used for learning processing. For example, the image data memory 10 stores a plurality of hyper-spectral image data of, for example, 256×256×31, which is learning data, that is, the image data of a hyper-spectral image of 31 bands.

[0040] [Camera Image Generation Simulator 20] The camera image generation simulator 20 is a compressed image generation model that generates the image data of a compressed image in which an input image, which is a hyper-spectral image, is compressed based on parameters corresponding to the optical element 212 having a fine structure.

[0041] The camera image generation simulator 20 is, for example, an image generation simulator that infers (generates) an image captured by a camera by simulation. For example, the camera image generation simulator 20 is a mathematical model for simulating an image captured by an optical element 212 having a fine structure. Note that any model can be adopted as long as the camera image generation simulator 20 is a model in a form that can be processed by a computer and a model whose parameters can be updated by learning processing.

[0042] The camera image generation simulator 20 can generate image data with a reduced number of bands in response to an input of hyper-spectral image data of dozens of bands or more. Hereinafter, the case where the image data of a hyper-spectral image with 31 bands is input to the camera image generation simulator 20 will be described as an example. Note that the number of bands of the hyper-spectral image in which image data is input to the camera image generation simulator 20 is not limited to 31 bands, and may be any number of bands, and may be dozens or hundreds of bands.

[0043] The camera image generation simulator 20 generates image data of an image with a reduced number of bands of a 31-band hyperspectral image in response to an input of 256×256×31 hyperspectral image data. For example, the camera image generation simulator 20 generates image data of a compressed image in which the number of bands of a 31-band hyperspectral image is compressed to 3 bands in response to an input of 256×256×31 hyperspectral image data.

[0044] Note that the image generated by the camera image generation simulator 20 is not limited to a 3-band (RGB) color image, and the imaging element 211 may be a 1-band (monochrome) image. For example, when the imaging unit 210 captures an image of 256×256×1 (monochrome image), the camera image generation simulator 20 may output a 1-band (monochrome) image.

[0045] The camera image generation simulator 20 uses a plurality of parameters corresponding to fluctuations in the relative position of the optical element 212 with respect to the imaging element 211 to generate image data of a plurality of compressed images in which the image data of the input image, which is a hyperspectral image, is compressed respectively. The fluctuation in the relative position of the optical element 212 with respect to the imaging element 211 is a fluctuation from the reference of the relative position of the optical element 212 with respect to the imaging element 211, and for example, is a fluctuation due to an assumed mounting error of the optical element 212 to the housing 213. The reference of the relative position of the optical element 212 with respect to the imaging element 211 is a preset reference position.

[0046] [Reconstruction processing model 30] As described above, the reconstruction processing model 30 is a model (machine learning model) that takes, as an input, the image data of the compressed image CI, which is the captured image captured by the imaging element 211, and outputs the image data of the output image, which is a hyperspectral image having a larger number of bands (channels) than the compressed image CI, by performing a reconstruction process on the input image data.

[0047] The reconstruction processing model 30 executes a reconstruction process for reconstructing an image based on a matrix (for example, an observation matrix Φ) defined by the imaging process of the optical element 212 and the image data of an image (observation image) in which the PSF of each wavelength is convolved.

[0048] For example, the reconstruction process can be formulated as a process that solves an optimization problem with respect to the observation matrix Φ defined by the optical system and the compressed image g, which is the acquired encoded image, as shown in equation (1) below.

[0049]

[0050] In equation (1), the first term f on the right-hand side represents the image we want to reconstruct (such as a hyperspectral image). Here, since the number of data points in the observed image is significantly less than the number of data points in the image we want to reconstruct (reconstructed image), there are infinitely many solutions that satisfy Φf - g = 0. However, by adding a regularization term as the second term, it may be easier to find the most plausible image as the reconstructed image (reconstructed image^f). Note that the second term (regularization term) does not have to be included in equation (1). Also, when "^f" is written for f, it is equivalent to "a symbol with "^" written directly above "f".

[0051] Regarding regularization terms, various types have been proposed for spectral images, and in this embodiment, any of these regularization terms can be applied. In the example of equation (1), R corresponds to the prior probability of the signal based on prior information (image-likeness), and it utilizes the sparsity that images generally possess, such as small differences between adjacent pixels. τ is a balancing parameter. In this embodiment, a regularization term called SSTV (Spatio-Spectral Total Variation) (Reference 1) is used, and it is optimized to minimize the differences between adjacent pixels in spatial and wavelength dimensions during image reconstruction. Reference 1: Aggarwal, H. K., & Majumdar, A. (2016). Hyperspectral image denoising using spatio-spectral total variation. IEEE Geoscience and Remote Sensing Letters, 13(3), 442-446.

[0052] In this way, the reconstruction processing model 30 generates hyperspectral image data by performing a process to reconstruct the spatial and spectral information of the subject from the observed image data, which has been subjected to different convolution operations for each wavelength, based on compressed sensing. Compressed sensing is a technique for recovering the target signal from a small number of samples.

[0053] If the observation process is known (for example, the PSF of the optical element 212 and the wavelength sensitivity characteristics of the sensor (image sensor 211, etc.)), the optically encoded image can have its real image information restored by performing appropriate signal processing using the reconstruction processing model 30. Therefore, the imaging device 200 performs signal processing using compressed sensing, a method that reconstructs (restores) the target with high accuracy from a small amount of information by utilizing the sparsity of natural images. Reconstruction refers to restoring the original signal from the encoded signal. Because the optical element 212 has a wavelength-dependent PSF, the imaging device 200 can perform different encoding for each wavelength component of the real image, and therefore, by performing image reconstruction processing based on compressed sensing using the reconstruction processing model 30, the hyperspectral image can be restored.

[0054] [Error Calculator 40] The error calculator 40 functions as a processing unit that performs error calculations during the learning process. The error calculator 40 calculates the root mean squared error (RMSE) between the image data of the image input from the image data memory 10 to the camera image generation simulator 20 and the image data of the image output from the reconstruction processing model 30 during the learning process.

[0055] For example, the error calculator 40 calculates the root mean square error (RMSE) between the hyperspectral image data input to the camera image generation simulator 20 and the hyperspectral image data output from the reconstruction processing model 30. Note that the root mean square error is just one example of an error, and the error is not limited to the root mean square error; any other index such as mean square error or mean absolute error may be used.

[0056] [Acquisition Unit 50] The acquisition unit 50 acquires image data of multiple compressed images corresponding to the variation in the relative position of the optical element 212 with respect to the image sensor 211 located downstream of the optical element 212 which has a microstructure.

[0057] For example, the acquisition unit 50 can acquire image data of multiple compressed images generated by the camera image generation simulator 20 using a number of parameters corresponding to the number of changes in the relative position of the optical element 212 with respect to the image sensor 211, as image data of multiple compressed images corresponding to the changes in the relative position of the optical element 212 with respect to the image sensor 211.

[0058] Furthermore, the acquisition unit 50 can also acquire multiple image data, each of which is output from the image sensor 211 when the relative positions of the optical elements 212 with respect to the image sensor 211 are different in the imaging unit 210 shown in Figure 2, as multiple compressed image data corresponding to the variation in the relative position of the optical elements 212 with respect to the image sensor 211. In this way, the acquisition unit 50 can acquire multiple compressed image data from the image sensor 211 and the like that have actually been created.

[0059] Furthermore, the acquisition unit 50 acquires hyperspectral image data obtained from multiple images of imaging targets captured by the image sensor 211 in the imaging unit 210 shown in Figure 2, from an external device or the like. The acquisition unit 50 stores the hyperspectral image data acquired from the external device or the like in the image data memory 10 as multiple hyperspectral image data used in the learning process.

[0060] [Learning Unit 60] The learning unit 60 performs model learning processing. For example, the learning unit 60 performs learning processing to generate a reconstruction processing model 30 using image data of multiple compressed images acquired by the acquisition unit 50. As described above, the reconstruction processing model 30 is a model that reconstructs hyperspectral image data from image data of images obtained by the image sensor 211.

[0061] The learning unit 60 controls the learning process to learn such that the value of the loss function, which includes elements related to the error between the image data of the input image and the image data of the output image (also called the "first element"), such as the root mean square error from the error calculator 40, is minimized. In this way, the image processing system 1 performs optimization processing of the reconstruction processing model 30 by learning processing using a loss function that includes the first element.

[0062] Figure 4 is a diagram illustrating the overview of the learning process according to the embodiment. As shown in Figure 4, the camera image generation simulator 20 generates compressed image data g by compressing the image data of the input image f, which is a hyperspectral image, using the observation matrix Φ. For example, the camera image generation simulator 20 is a model that, when image data of the input image f is input, derives compressed image data g (=Φf) by performing calculations using the image data of the input image f and the observation matrix Φ.

[0063] Thus, the camera image generation simulator 20 is an image generation simulator that generates a compressed image, which is a simulation result of an image captured using the optical element 212, by converting it into an image with fewer bands than a hyperspectral image using parameters corresponding to the optical element 212.

[0064] Here, the observation matrix Φ is a parameter corresponding to the imaging unit 210. That is, the observation matrix Φ is a parameter corresponding to the PSF of the optical element 212. The learning unit 60 sequentially sets a plurality of parameters in the camera image generation simulator 20 that correspond to the variation in the relative position of the optical element 212 with respect to the image sensor 211, as parameters corresponding to the PSF of the optical element 212. As a result, the learning unit 60 can sequentially output image data of a plurality of compressed images g from the camera image generation simulator 20, each representing a state in which the relative position of the optical element 212 with respect to the image sensor 211 is varied in different ways.

[0065] The learning unit 60 has calculation formulas (e.g., transformation formulas or transformation matrices) that generate parameters corresponding to each of the following: rotational displacement, center displacement, tilt, and Z-direction displacement, as well as combinations thereof, from parameters when the relative position of the optical element 212 with respect to the image sensor 211 is at a reference position. The learning unit 60 calculates a number of parameters corresponding to the variation in the relative position of the optical element 212 with respect to the image sensor 211 and sequentially sets them in the camera image generation simulator 20.

[0066] The amount of each type of deviation, such as rotational deviation, center deviation, tilt, and Z-direction deviation, is pre-set in the learning unit 60, for example, but is not limited to this example. For example, the user can set these values ​​in the learning unit 60 via an input unit (not shown).

[0067] Furthermore, multiple parameters corresponding to variations in the relative position of the optical element 212 with respect to the image sensor 211 may be pre-set in the camera image generation simulator 20. In this case, the camera image generation simulator 20 can also output compressed image data g by using the multiple parameters randomly or in an order according to a predetermined rule.

[0068] Furthermore, as shown in Figure 4, the reconstruction processing model 30 generates image data for the reconstructed image ^f, which is a hyperspectral image, from the image data for the compressed image g. In this way, during the learning process, the reconstruction processing model 30 receives the image data for the compressed image generated by the camera image generation simulator 20 via the acquisition unit 50 (see Figure 1), and outputs image data for the output image, which is a hyperspectral image with a larger number of bands than the input compressed image.

[0069] Figure 4 shows an example in which compressed image data generated by the camera image generation simulator 20 is used. However, the compressed image data may be compressed image data generated by an imaging device 200 or the like. In other words, the compressed image data may be compressed image data generated using the optical element 12 that was actually fabricated.

[0070] The reconstruction processing model 30 performs a generation process to generate image data for multiple reconstructed images ^f from the image data of multiple compressed images g acquired by the acquisition unit 50. For example, the reconstruction processing model 30 performs a generation process to generate image data for reconstructed images ^f from the image data of the compressed images g acquired by the acquisition unit 50 for each compressed image g acquired by the acquisition unit 50. Such a generation process is performed, for example, for each image data of a hyperspectral image stored in the image data memory 10.

[0071] For example, the reconstruction processing model 30 performs a generation process to generate image data for multiple reconstructed images ^f from image data of multiple compressed images g generated by the camera image generation simulator 20 and acquired by the acquisition unit 50, using multiple parameters corresponding to the variation in the relative position of the optical element 212 with respect to the image sensor 211. The generation of image data for reconstructed images ^f is performed for each image data of compressed images g. For example, the reconstruction processing model 30 performs a generation process to generate image data for reconstructed images ^f from image data of compressed images g generated by the camera image generation simulator 20 and acquired by the acquisition unit 50, using one parameter corresponding to the variation in the relative position of the optical element 212 with respect to the image sensor 211, for each of a number of multiple parameters corresponding to the number of variations in the relative position of the optical element 212 with respect to the image sensor 211.

[0072] Furthermore, the reconstruction processing model 30 performs a generation process to generate image data for multiple reconstructed images ^f from image data of multiple compressed images g acquired by the acquisition unit 50 from an external device or the like. The generation of image data for reconstructed images ^f is performed for each image data of compressed images g. For example, the reconstruction processing model 30 performs a generation process to generate image data for reconstructed images ^f from image data of compressed images g acquired by the acquisition unit 50 from an external device or the like, for each image data of compressed images g acquired by the acquisition unit 50 from an external device or the like.

[0073] The learning unit 60 performs a learning process on the reconstruction processing model 30. For example, the learning unit 60 performs an optimization process to update the reconstruction processing model 30 by performing a learning process that learns the reconstruction processing model 30 using the loss function described above (which may be referred to as the loss function L below). For example, in the learning process, the hyperspectral image data stored in the image data memory 10 is used in the learning process as the input image data and ground truth information (ground truth image data) of the camera image generation simulator 20.

[0074] The learning unit 60 performs optimization processing of the reconstruction processing model 30 by a learning process using a loss function L that includes a first element relating to the error between the input image data and the output image data. Based on the hyperspectral image data stored in the image data memory 10 and the hyperspectral image data output from the reconstruction processing model 30, the learning unit 60 performs a learning process (optimization process) to adjust the parameters of the reconstruction processing model 30 so that the value of the loss function L, which includes a first element relating to the error between the input image data and the output image data, is minimized.

[0075] For example, the learning unit 60 performs learning processing using methods such as backpropagation so that the image data of the reconstructed image ^f, which is the result of the reconstruction processing model 30 performing reconstruction processing on the compressed image data output by the camera image generation simulator 20 that receives the image data of the input image f, approaches the image data of the input image f. Note that the learning method used by the learning unit 60 for the learning processing of the reconstruction processing model 30 can be any learning method as long as the reconstruction processing model 30 is capable of learning, and a detailed explanation is omitted.

[0076] Furthermore, the learning unit 60 can also optimize the parameters corresponding to the PSF of the optical element 212 when the relative position of the optical element 212 with respect to the image sensor 211 is the reference position. The physical configuration of the optical element 212 (such as the structure 214) can be designed from the optimized parameters corresponding to the PSF of the optical element 212.

[0077] The learning unit 60 can optimize the parameters of the compressed image generation model in addition to the parameters of the reconstruction processing model 30 by, for example, changing the parameters of the reconstruction processing model 30 and the parameters of the compressed image generation model, and by performing learning processing on the hyperspectral image generation model including the compressed image generation model (camera image generation simulator 20) and the reconstruction processing model 30. Subsequently, the learning unit 60 optimizes the parameters of the compressed image generation model using a plurality of parameters corresponding to the variation in the relative position of the optical element 212 with respect to the image sensor 211.

[0078] Furthermore, the learning unit 60 can also optimize the parameters of the compressed image generation model for each reference position of the optical element 212 while changing the reference position of the optical element 212 relative to the image sensor 211. Subsequently, the learning unit 60 optimizes the parameters of the compressed image generation model for each reference position using a plurality of parameters corresponding to the variation in the relative position of the optical element 212 relative to the image sensor 211. In this case, for example, the learning unit 60 can determine the parameters of the hyperspectral image generation model at the reference position with the smallest loss function value obtained in the learning process including variation from the reference position, among a plurality of mutually different reference positions, as the parameters of the optical element 212 and the parameters of the reconstruction processing model 30.

[0079] In this way, the camera image generation simulator 20 generates multiple compressed image data, each of which is compressed image data of the input image, which is a hyperspectral image, using multiple parameters corresponding to the variation in the relative position of the optical element 212 with respect to the image sensor 211. Then, the learning unit 60 generates a reconstruction processing model that reconstructs the hyperspectral image data from the image data of the image obtained by the image sensor 211 using the multiple compressed image data generated by the camera image generation simulator 20. As a result, the image processing system 1 can improve the accuracy of the reconstruction processing in the imaging device 200, which is an example of a compressed sensing type hyperspectral camera.

[0080] [Example of Information Processing Flowchart] From here, an example of a flowchart of the learning process executed by the image processing system 1 will be described. Figure 5 is a flowchart of the learning process procedure according to the embodiment. The flowchart in Figure 5 starts, for example, when an input instructing the start of the learning process is received. For example, when the image processing system 1 receives an instruction to generate a reconstruction processing model 30, it starts the learning process using the learning data stored in the image data memory 10.

[0081] As shown in Figure 5, the learning unit 60 controls the camera image generation simulator 20 and causes the camera image generation simulator 20 to generate multiple compressed image data, each of which is a hyperspectral image, using multiple parameters corresponding to variations in the relative position of the optical element 212 with respect to the image sensor 211 (step S10).

[0082] Next, the learning unit 60 generates a reconstruction processing model 30 that reconstructs hyperspectral image data from image data obtained by the image sensor 211 using the image data of the multiple compressed images generated in step S10 (step S11).

[0083] In the process of step S11, the learning unit 60 generates the reconstruction processing model 30 such that the difference between the image data of the hyperspectral image generated in the reconstruction processing model 30 and the image data of the hyperspectral image input to the camera image generation simulator 20 is minimized.

[0084] Next, the learning unit 60 outputs the reconstruction processing model 30 generated in step S11 (step S12). For example, the learning unit 60 may output the reconstruction processing model 30 generated in step S11 to a memory (not shown) for storage, or it may transmit the reconstruction processing model 30 generated in step S11 to an external device via a communication unit (not shown). Note that the processing in step S10 may be, for example, a process of acquiring image data of multiple captured images (compressed images) captured by the image sensor 211 while changing the relative position of the optical element 212 with respect to the image sensor 211 in the imaging device 100. In this case, the learning unit 60 generates a reconstruction processing model 30 that reconstructs hyperspectral image data from the image data of multiple captured images (compressed images) acquired from the imaging device 100.

[0085] [Other] The image processing system 1 according to this embodiment may also include the imaging device 200 shown in Figure 2. Figure 6 shows another example of the configuration of the image processing system 1 according to this embodiment. The image processing system 1 shown in Figure 6 includes an imaging unit 210, an image data memory 10, a camera image generation simulator 20, an image processing unit 220 having a reconstruction processing model 30, an error calculator 40, a controller 70 including a learning unit 60, an image data memory 230, a display 240, and switches SW1 and SW2. Switch SW1 functions as the acquisition unit 50 shown in Figure 1.

[0086] The controller 70 switches between learning and inference processing by switching switches SW1 and SW2. For example, when performing learning processing, the controller 70 sets switches SW1 and SW2 to contact T (upper in Figure 6) and sets the processing flow so that the output from the camera image generation simulator 20 is input to the reconstruction processing model 30, and the output from the reconstruction processing model 30 is input to the error calculator 40. In the learning process, the compressed image data output from the camera image generation simulator 20 is output to the reconstruction processing model 30 via switch SW1. Also in the learning process, the hyperspectral image data output from the reconstruction processing model 30 is output to the error calculator 40 via contact T of switch SW2.

[0087] Furthermore, when the controller 70 performs learning processing, it can also set switch SW1 to contact I (lower side in Figure 6) and switch SW2 to contact T (upper side in Figure 6), so that the output from the image sensor 211 of the imaging unit 210 is input to the reconstruction processing model 30, and the output from the reconstruction processing model 30 is input to the error calculator 40. In the learning process, the compressed image data output from the image sensor 211 of the imaging unit 210 is output to the reconstruction processing model 30 via switch SW1. Also in the learning process, the hyperspectral image data output from the reconstruction processing model 30 is output to the error calculator 40 via contact T of switch SW2.

[0088] Furthermore, when the controller 70 performs inference processing, it sets switches SW1 and SW2 to contact I (lower side in Figure 6), and sets the processing flow so that the image captured by the imaging unit 210 is input to the reconstruction processing model 30, and the output from the reconstruction processing model 30 is input to the image data memory 230. For example, in inference processing, the image data of the compressed 3D color space image output from the image sensor 211 is output to the trained reconstruction processing model 30 via switch SW1. Also in inference processing, the image data of the hyperspectral image output from the reconstruction processing model 30 is output to contact I of switch SW2 and to the image data memory 230.

[0089] Thus, the image processing system 1 shown in Figure 6 can perform the functions of the imaging device 200. Note that the image processing system 1 shown in Figure 6 may be configured with the imaging unit 210 and display 240 and the other elements as separate components. For example, the image processing system 1 may have an image data memory 10, an image processing unit 220 having a camera image generation simulator 20 and a reconstruction processing model 30, a controller 70 including an error calculator 40 and a learning unit 60, an image data memory 230, and switches SW1 and SW2, all implemented by a server.

[0090] Furthermore, as mentioned above, the loss function used in the learning process includes a first element relating to the error between the image data of the input image and the image data of the output image. However, the first element is merely one example of the elements included in the loss function, and the image processing system 1 may use a loss function that includes various elements in addition to the first element to perform the learning process of the hyperspectral image generation model, including the reconstruction processing model 30.

[0091] The loss function L may include an element (also called the "second element") relating to the maximum value of the response characteristics of the optical element 212 for each wavelength. For example, the loss function L may include a second element relating to the maximum value of the PSF for each wavelength of the optical element 212. In this case, the image processing system 1 performs an optimization process of the hyperspectral image generation model, including the reconstruction processing model 30, by a learning process using the loss function L including the second element.

[0092] For example, if the image processing system 1 performs the learning process taking into account the second element, a function relating to the second element, as shown in equation (2), may be added to the loss function L.

[0093]

[0094] In equation (2), x and y represent the coordinates of the sensor surface, and λ represents the wavelength. Equation (2) is formulated with p > 1, PSF = P(x, y, λ), and the center = (x, y) = (0, 0).

[0095] Note that equation (2) is merely an example; for instance, if the image processing system 1 performs the learning process taking into account the second element, a function relating to the second element, as shown in equation (3), may be added to the loss function L.

[0096]

[0097] In equation (3), x and y represent the coordinates of the sensor surface, and λ represents the wavelength. Equation (3) is formulated with r as the limit value (radius) of the PSF size, P(x,y,λ) as the PSF, and (x,y) = (0,0) as the center. By performing a learning process using equations (2), (3), etc., the image processing system 1 can add a limit to the maximum PSF size to the loss function, thereby reducing the computational cost during inference by reducing the PSF size (corresponding to the convolution kernel).

[0098] Furthermore, the loss function L may include an element (also called the "third element") relating to the homogenization of the response characteristics of the optical element 212 for each wavelength. For example, the loss function L may include a third element relating to the homogenization of the PSF of the optical element 212 for each wavelength. In this case, the image processing system 1 performs an optimization process of the hyperspectral image generation model, including the reconstruction processing model 30, by a learning process using the loss function L including the third element.

[0099] For example, if the image processing system 1 performs the learning process taking into account the third element, a function relating to the third element, as shown in equation (5) based on equation (4), may be added to the loss function L.

[0100]

[0101]

[0102] In equations (4) and (5), x and y represent the coordinates of the sensor surface, and λ represents the wavelength. Equations (4) and (5) are formulated with p > 1, q < 1, PSF is P(x, y, λ), and the center is (x, y) = (0, 0). By performing a learning process using equation (5) and the like, the image processing system 1 can add a loss function that equalizes the PSF size across the wavelength, thereby equalizing the spatial resolution.

[0103] The loss function L may include an element relating to the transmittance of the optical element 212 (also called the "fourth element"). For example, the loss function L may include a fourth element relating to the total transmittance of the optical element 212. In this case, the image processing system 1 performs an optimization process of the hyperspectral image generation model, including the reconstruction processing model 30, by a learning process using the loss function L including the fourth element.

[0104] For example, if the image processing system 1 performs the learning process taking into account the fourth element, a function relating to the fourth element, as shown in equation (6), may be added to the loss function L.

[0105]

[0106] Equation (6) is formulated with T as the average transmittance across space and wavelength at the lens surface (pupil surface). By performing a learning process using equation (6) and the like, the image processing system 1 can add the total lens transmittance to the loss function, thereby improving efficiency and mitigating stray light.

[0107] The image processing system 1 performs a training process for a hyperspectral image generation model, including the reconstruction processing model 30, using a loss function that includes any combination of the first to fourth elements described above. For example, the image processing system 1 performs a training process for a hyperspectral image generation model, including the reconstruction processing model 30, using a loss function that includes at least the first element.

[0108] For example, the image processing system 1 performs a training process for a hyperspectral image generation model including a reconstruction processing model 30 using a loss function that includes multiple elements. For example, the image processing system 1 performs a training process for a hyperspectral image generation model including a reconstruction processing model 30 using a loss function that includes at least one element from the second to fourth elements and the first element.

[0109] Then, an optical element 212 having response characteristics such as PSF corresponding to the observation matrix Φ of the camera image generation simulator 20 after the training process is created. An optical element 212 having PSF corresponding to the observation matrix Φ of the camera image generation simulator 20 after the training process is created. In this way, the optical element 212 is designed based on the parameters updated by the optimization process. The optical element 212 is shaped based on the response characteristics such as PSF transformed by the parameters updated by the optimization process. Then, the optical element 212 created based on the PSF corresponding to the observation matrix Φ of the camera image generation simulator 20 after the training process is implemented in the camera image generation simulator 20.

[0110] [Effects] As described above, the image processing system 1 of this embodiment comprises an acquisition unit 50 and a learning unit 60. The acquisition unit 50 acquires image data of multiple compressed images corresponding to fluctuations in the relative position of the optical element 212 with respect to the image sensor 211 located downstream of the optical element 212 having a microstructure. The learning unit 60 uses the image data of the multiple compressed images acquired by the acquisition unit 50 to generate a reconstruction processing model that reconstructs hyperspectral image data from the image data of the image obtained by the image sensor 211. As a result, the image processing system 1 can suppress a decrease in the accuracy of the hyperspectral image reconstruction process even when there is an error in the mounting of the optical element 212.

[0111] Furthermore, the image processing system 1 of this embodiment includes a camera image generation simulator 20 that generates compressed image data using a plurality of parameters corresponding to fluctuations in the relative position of the optical element 212. For example, the camera image generation simulator 20 includes a compressed image generation model that generates compressed image data from the input image data, which is a hyperspectral image, based on parameters corresponding to the optical element 212. The learning unit 60 performs an optimization process to update the hyperspectral image generation model, which includes the compressed image generation model and the reconstruction processing model, using a loss function that includes an element relating to the error between the input image data and the output image data output from the reconstruction processing model. As a result, the image processing system 1 can update (optimize) both the parameters corresponding to the optical element 212, i.e., the settings for the physical configuration of the optical element 212, and the parameters of the reconstruction processing model 30, i.e., the settings for the reconstruction process (information processing). For example, by creating the optical element 212 based on setting values ​​such as PSF converted from the parameters updated by the optimization process, an optical element 212 having a physical configuration suitable for generating a hyperspectral image in cooperation with the reconstruction processing model 30 is created. Thus, the image processing system 1 can appropriately support the design of a compressed sensing type hyperspectral camera.

[0112] [Program] A program can also be created that describes the processing performed by the computer included in the image processing system 1 according to the above embodiment in a language that can be executed by a computer. In one embodiment, the image processing system 1 can be implemented by installing an information processing program, which includes an image processing program that performs the above image processing, as packaged software or online software, on a desired computer. For example, by having the image processing system 1 execute the image processing program included in the above information processing program, the information processing performed by the image processing system 1 (for example, the processing of the camera image generation simulator 20, the processing of the reconstruction processing model 30, the processing of the error calculator 40, the processing of the acquisition unit 50, the processing of the learning unit 60, the processing of the controller 70, etc.) can be realized.

[0113] Figure 7 shows an example of a computer that executes an information processing program in the image processing system 1 according to the embodiment. The computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0114] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1031. The disk drive interface 1040 is connected to the disk drive 1041. The disk drive 1041 is used to insert a removable storage medium, such as a magnetic disk or an optical disk. The serial port interface 1050 is used to connect, for example, a mouse 1051 and a keyboard 1052. The video adapter 1060 is used to connect, for example, a display 1061.

[0115] Here, the hard disk drive 1031 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. The information described in the above embodiment is stored, for example, in the hard disk drive 1031 or the memory 1010.

[0116] Furthermore, the information processing program is stored in the hard disk drive 1031 as a program module 1093 containing instructions to be executed by the computer 1000, for example. Specifically, the program module 1093 containing instructions for each process executed by the image processing system 1 described in the above embodiment is stored in the hard disk drive 1031.

[0117] Furthermore, the data used for information processing by the information processing program is stored as program data 1094, for example, in the hard disk drive 1031. The CPU 1020 then reads the program module 1093 and program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as needed and executes the procedures described above.

[0118] Furthermore, the program module 1093 and program data 1094 related to the information processing program are not limited to being stored in the hard disk drive 1031; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1041 or the like. Alternatively, the program module 1093 and program data 1094 related to the information processing program may be stored in another computer connected via a network such as a LAN (Local Area Network) or WAN (Wide Area Network) and read by the CPU 1020 via a network interface 1070.

[0119] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in this embodiment. That is, other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are all included in the scope of the present invention.

[0120] 1 Image processing system 2 Image target 10,230 Image data memory 20 Camera image generation simulator 30 Reconstruction processing model 40 Error calculator 50 Acquisition unit 60 Learning unit 70 Controller 200 Imaging device 210 Imaging unit 211 Image sensor 212 Optical element 213 Housing 214 Structure 215 Transparent substrate 220 Image processing unit 240,1061 Display

Claims

1. An image processing system comprising: an acquisition unit that acquires image data of a plurality of compressed images corresponding to variations in the relative position of an optical element having a microstructure with respect to an image sensor located downstream of the optical element; and a learning unit that generates a reconstruction processing model for reconstructing hyperspectral image data from image data of an image obtained by the image sensor using the image data of the plurality of compressed images acquired by the acquisition unit.

2. The image processing system according to claim 1, further comprising a camera image generation simulator that generates image data of the plurality of compressed images using a plurality of parameters corresponding to the variation in the relative position of the optical elements.

3. The image processing system according to claim 2, characterized in that the learning unit sequentially sets the plurality of parameters in the camera image generation simulator and causes the camera image generation simulator to sequentially generate image data of the plurality of compressed images.

4. The image processing system according to claim 2 or 3, wherein the camera image generation simulator includes a compressed image generation model that generates compressed image data, which is obtained by compressing an input image that is a hyperspectral image, based on parameters corresponding to the optical elements, and the learning unit performs an optimization process to update the hyperspectral image generation model, which includes the compressed image generation model and the reconstruction processing model, using a loss function that includes an element relating to the error between the input image and the output image output from the reconstruction processing model.

5. An image processing method characterized by comprising: an acquisition step of acquiring image data of a plurality of compressed images corresponding to variations in the relative position of an optical element having a microstructure with respect to an image sensor located downstream of the optical element; and a learning step of generating a reconstruction processing model that reconstructs hyperspectral image data from an image obtained by the image sensor using the image data of the plurality of compressed images acquired in the acquisition step.

6. An image processing program characterized by causing a computer to execute an acquisition procedure for acquiring image data of multiple compressed images corresponding to variations in the relative position of an optical element having a microstructure with respect to an image sensor located downstream of the optical element; and a learning procedure for generating a reconstruction processing model that uses the image data of the multiple compressed images acquired in the acquisition procedure to reconstruct hyperspectral image data from an image obtained by the image sensor.

Images