Imaging device

Abstract

An imaging device (10) comprises: an optical element (12) including a first lens that receives visible light and a second lens that receives invisible light; and an imaging element (11) that receives light arriving via the optical element (12). The first lens and the second lens are arranged in substantially the same plane.

Description

Imaging device

[0001] The present invention relates to an imaging device.

[0002] In recent years, research on compressive sensing type hyperspectral cameras has been underway (see Patent Documents 1 and 2). For example, a compressive sensing type hyperspectral camera performs optical encoding (operation / arithmetic of optical information) on an imaging target and captures an image, and restores spatial information and wavelength information using a reconstruction process that utilizes the sparsity of natural images. In addition, techniques for improving the accuracy of the reconstruction process in a compressive sensing type hyperspectral camera have been provided (see Patent Document 3).

[0003] International Publication No. 2022 / 162801, International Publication No. 2022 / 162800, International Publication No. 2021 / 234797

[0004] In the above prior art, a hyperspectral image that captures more color information (wavelengths) than a color image that mimics the human eye is captured using a metalens having a plurality of fine structures. In a hyperspectral image, the properties of a subject that are difficult to grasp with the human eye can be distinguished. Since a hyperspectral image has characteristics significantly different from those of a color image that can be grasped by the human eye, a method of autofocus for a color image cannot be directly used for autofocus of a hyperspectral image. On the other hand, in manual focus where the focus of a hyperspectral image is manually adjusted, the usability deteriorates and the range of use is limited. Therefore, in imaging a hyperspectral image, there is room for improvement in terms of autofocus control, and autofocus control using an optical element having a plurality of fine structures is desired.

[0005] The present invention has been made in view of the above, and an object thereof is to enable autofocus control using an optical element having a plurality of fine structures.

[0006] To solve the aforementioned problems and achieve the objective, the system includes an optical element comprising a first lens for receiving visible light and a second lens for receiving invisible light, and an image sensor for receiving light that has reached through the optical element. The first lens and the second lens are arranged on substantially the same plane.

[0007] According to the present invention, autofocus control using an optical element having multiple microstructures is made possible.

[0008] Figure 1 shows an example of an image processing system according to the embodiment. Figure 2 shows an example of the configuration of a device that performs information processing according to the embodiment. Figure 3 shows an example of a model information storage unit according to the embodiment. Figure 4 is a flowchart of the focus control processing procedure. Figure 5 shows an example of an image captured by an imaging device that performs focus control processing. Figure 6 is a diagram for explaining the focus control processing procedure. Figure 7 is a flowchart of the inference processing procedure. Figure 8 is a diagram for explaining the overview of the learning process. Figure 9 shows an example of a computer that executes an information processing program.

[0009] Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited by this embodiment. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0010] <Embodiment> <Configuration and Processing of Image Processing System> Hereinafter, an example of the configuration of the image processing system 1 according to the embodiment and the processing performed by the image processing system 1 will be described with reference to Figure 1. Figure 1 is a diagram showing an example of the image processing system 1 according to the embodiment. For example, Figure 1 is a block diagram showing an example of the configuration of the image processing system 1 according to the embodiment.

[0011] First, an example of the configuration of the image processing system 1 will be explained using Figure 1, and then an overview of the inference processing performed by the image processing system 1 will be described. As shown in Figure 1, the image processing system 1 includes an imaging device 10 and a reconstruction processing model 20. Note that the configuration of the image processing system 1 shown in Figure 1 is merely an example, and the image processing system 1 can be configured in any way that allows it to perform the desired processing.

[0012] <Imaging Device> The imaging device 10 functions as an imaging unit that captures images. The imaging device 10 includes an image sensor 11 and an optical element 12. The image sensor 11 and the optical element 12 are arranged facing each other. The image sensor 11 is a solid-state image sensor such as a CCD image sensor or a CMOS image sensor, and is composed of a photoelectric conversion element and circuit elements such as transistors provided around it. The optical element 12 is an optical member having a microstructure. For example, the optical element 12 is a metasurface (metalens), but the details will be described later. In addition, although not shown in Figure 1, the imaging device 10 has a signal processing unit that processes the photoelectric conversion signal output from the image sensor 11 to generate an image signal (corresponding to the compressed image CI in Figure 1).

[0013] As shown in Figure 1, in the imaging device 10, light such as natural light or illumination light is shone onto the object to be imaged (actual image, subject), and the light transmitted / reflected / scattered by the object to be imaged 2, or the light emitted from the object to be imaged 2, forms an optical image on the image sensor 11 by the optical element 12.

[0014] As described above, the optical element 12 having a microstructure has a plurality of fine structures 14 arranged on the same plane. The optical element 12 may also have a transparent substrate 13. In Figure 1, the plurality of structures 14 are shown as an example in which they are arranged on the transparent substrate 13 in the plane direction of the transparent substrate 13, but the plurality of structures 14 may also be arranged within the transparent substrate 13 in the plane direction of the transparent substrate 13.

[0015] For example, the optical element 12 consists of a fine binary structure. In the example shown in Figure 1, the plurality of structures 14 of the optical element 12 include a plurality of first structures 14a and a plurality of second structures 14b. The plurality of first structures 14a and the plurality of second structures 14b have a columnar shape with their longitudinal direction intersecting (e.g., orthogonal to) the plane direction of the transparent substrate 13, and are arranged at predetermined intervals.

[0016] The image sensor 11 receives light that has arrived via optical elements. For example, the image sensor 11 has multiple pixels arranged in a two-dimensional array, each containing a photoelectric conversion element. An example of a photoelectric conversion element is a photodiode (PD). Each pixel corresponds to red (R), green (G), and blue (B). An example of the wavelength band of red light is the wavelength λ. ０ Therefore, 600 nm < λ ０ ≤ 800 nm. An example of the wavelength range for green light is 500 nm < λ ０ It is ≤600 nm. An example of the wavelength range of blue light is λ ０ The wavelength is ≤500 nm. Pixels R, G, and B may be in a Bayer array. Alternatively, the pixels may be for monochrome images.

[0017] The incident light reaches the image sensor 11 via the optical element 12. The charge generated at each pixel of the image sensor 11 is converted into an electrical signal that forms the basis of the pixel signal by a transistor or the like (not shown). Note that the process of converting light into an electrical signal and generating digital data, which is image data (also simply called "image"), is a known technique, so a detailed explanation is omitted.

[0018] The optical element 12 is positioned on the side into which light from the imaging target 2 is incident. For example, the optical element 12 is positioned so as to cover the image sensor 11 when viewed from above (front view). The optical element 12 is constructed by arranging a plurality of structures 14 on the bottom surface of a transparent substrate 13, for example, periodically (having a periodic structure). The transparent substrate 13 is, for example, SiO ２ This is a low refractive index transparent substrate made of materials such as (refractive index n = 1.45).

[0019] For example, the optical element 12 is composed of a plurality of structures 14, each having a width less than or equal to the wavelength of light when viewed from above. For example, the optical element 12 has a rotationally symmetric shape (e.g., a four-fold rotationally symmetric shape) when viewed from above (in cross-section). The optical element 12 can employ any structure as long as it has the desired characteristics, and may have a two-dimensional structure or a three-dimensional structure. The optical element 12 can control the phase and light intensity according to the characteristics of light (wavelength, polarization, angle of incidence) simply by changing the parameters of the structures 14.

[0020] The optical element 12 has different PSF (Point Spread Function) depending on the wavelength, and its imaging characteristics (degree of blurring) differ depending on the wavelength of light from the imaging target 2. The light from the imaging target 2 is imaged onto the image sensor 11 by the optical element 12 with wavelength-dependent PSF function and acquired as an image (RGB image or monochrome image).

[0021] The image captured by the imaging device 10 having the optical element 12 described above (compressed image CI, etc.) corresponds to the result of optical convolution calculation performed on the imaging target 2 (real image) for each wavelength by the wavelength-dependent PSF of the optical element 12, and integration along the wavelength dimension on the pixel. The optical element 12 and the image sensor 11 capture (acquire) an image in an optically encoded and compressed state. In the case of the image sensor 11 being a color image sensor, after the convolution calculation, multiplication is performed according to the wavelength sensitivity of each R, G, and B pixel of the image sensor 11, and then integration along the wavelength dimension is performed on the pixel. In this way, the imaging device 10 forms an optically encoded image on the image sensor 11 using a single optical element 12. In other words, the imaging device 10 can perform effective encoding in spectral image reconstruction using a single optical element 12.

[0022] The imaging device 10 converts the light incident on the image sensor 11 into, for example, 256 × 256 × 3 image data (electrical signals) and outputs it. In the notation of the image data, x × y represents x pixels × y pixels, and x × y × z (for example, x, y, and wavelength in Figure 1) represents x pixels × y pixels × z bands (channels). For example, if z is 3, it becomes 3-band color image data. For example, when capturing a monochrome image (z = 1), the image sensor 11 converts the incident light into, for example, 256 × 256 × 1 image data (electrical signals) and outputs it.

[0023] The multiple first structures 14a and multiple second structures 14b of the optical element 12 are arranged on the same plane. The optical distance from the multiple second structures 14b to the image sensor 11 is equal to the optical distance from the multiple first structures 14a to the image sensor 11. The transparent substrate 13 has a transparent substrate 13a and a transparent substrate 13b. The multiple first structures 14a are arranged on the transparent substrate 13a, and the multiple second structures 14b are arranged on the transparent substrate 13b. The transparent substrate 13a and the transparent substrate 13b may be a single unit or may be separate.

[0024] The multiple first structures 14a are configured to have a first response characteristic corresponding to the wavelength of visible light. For example, the first response characteristic is an optical characteristic that guides visible light from the light incident on the multiple first structures 14a to a responsive position (for example, a pixel 11a) on the image sensor 11. In the pixel 11a region, the image sensor 11 captures an image DI (hereinafter also referred to as the first image) by receiving visible light that has arrived via the multiple first structures 14a. The image sensor 11 captures an image that captures visible light due to the first response characteristics of the multiple first structures 14a. Therefore, the first image is an image that can be perceived by the human eye and does not include images that cannot be perceived by the human eye. The first image is, for example, a color image or a monochrome image.

[0025] Furthermore, the multiple second structures 14b are configured to have different second response characteristics depending on the wavelength in the wavelength band that includes the non-visible light wavelength band. For example, the second response characteristic is an optical characteristic that guides light including non-visible light incident on the multiple second structures 14b to a responsive position (for example, a pixel 11b) of the image sensor 11 with a different point spreading function for each wavelength. The wavelength band that includes the non-visible light wavelength band includes, for example, the wavelength band of light other than visible light (electromagnetic waves) such as X-rays, ultraviolet rays, and infrared rays. The wavelength band that includes the non-visible light wavelength band may or may not include the visible light wavelength band. For example, the wavelength band that includes the non-visible light wavelength band may be a wavelength band that extends from visible light to near-infrared light, etc.

[0026] For example, the multiple second structures 14b function as lenses with PSFs of distinctly different shapes depending on the wavelength (wavelength-dependent PSF lenses), generating images by applying different convolution operations to the real image (subject) for each wavelength. For example, if the multiple second structures 14b are wavelength-dependent PSF lenses, and an object is imaged using the multiple second structures 14b of this optical element 12, a convolution operation is performed on the real image with PSFs different for each wavelength, and the result is imaged on the image sensor 11. In this way, the image sensor 11 acquires an observed image in which different convolution operations have been performed for each wavelength by the multiple second structures 14b of the optical element 12, which are wavelength-dependent PSF lenses.

[0027] In the pixel 11b region, the image sensor 11 captures an image (hereinafter also referred to as the second image) by receiving invisible light or light containing invisible light that has arrived via a plurality of second structures 14b. The imaging device 10 (signal processing unit) generates a compressed image CI from the second image captured by a lens having a PSF realized by a plurality of second structures 14b, which can reconstruct a hyperspectral image. The second image is an image that captures more color information (wavelengths) than the image that captures visible light, and includes images that cannot be perceived by the human eye. Therefore, the compressed image CI generated from the second image is similarly an image that captures more color information (wavelengths) than the image that captures visible light, and includes images that cannot be perceived by the human eye. From the above, the second image and the compressed image CI have characteristics that are significantly different from the first image that captures visible light.

[0028] Therefore, the imaging device 10 cannot use known autofocus techniques to focus the second image or the compressed image CI. On the other hand, manual focus for hyperspectral images is less convenient and has a limited range of use. Specifically, with manual focus techniques, if the relative position of the imaging device 10 (camera) and the subject changes drastically, it is not possible to track this change and adjust the focus, resulting in a blurred image that cannot be used to generate a reconstructed image in subsequent processing. In other words, manual focus techniques can only be used when the distance between the imaging device 10 and the subject is fixed. In contrast, since the first image is an image that captures visible light, the imaging device 10 can use known autofocus techniques to focus the first image.

[0029] The imaging device 10 according to this embodiment includes an auxiliary lens in addition to a main lens in the optical element 12. The optical distance between the optical element 12 and the image sensor 11 is determined by the focal length of the lens, as in a normal imaging device. Therefore, the auxiliary lens is a lens with the same focal length as the main lens. As a result, the imaging device 10 uses the focus adjusted by the auxiliary lens as the focus of the main lens to adjust the focus of the main lens. The auxiliary lens may also be used as an achromatic lens. In one embodiment, a lens realized by a plurality of second structures 14b is used as the main lens, and a lens realized by a plurality of first structures 14a is used as the auxiliary lens, resulting in a two-lens configuration. Specifically, the imaging device 10 uses a known autofocus method to focus the first image captured by the auxiliary lens realized by the plurality of first structures 14a. As a result, the imaging device 10 identifies the position where the first image is in focus as the focus position, and uses that position as the position where the second image is in focus (focus position). In other words, at the focal position of the first image, the imaging device 10 generates a compressed image CI from the second image captured by the wavelength-dependent PSF lens realized by a plurality of second structures 14b, from which the hyperspectral image SI can be reconstructed.

[0030] The imaging device 10 has a moving mechanism 15 that moves at least one of the optical element 12 or the image sensor 11 so as to change the optical distance between the optical element 12 and the image sensor 11. Below, we will describe an autofocus method in which the imaging device 10 automatically controls the position in which the first image is in focus by controlling the moving mechanism 15 to move the optical element 12. Note that any known autofocus method, such as a contrast detection method, a phase difference detection method, or an infrared method, can be used for autofocus control.

[0031] The imaging device 10 may include known components such as an infrared light cut-off optical filter, an electronic shutter, a viewfinder, a power supply (battery), and a flashlight, but their descriptions are omitted as they are not particularly necessary for understanding the present invention. Furthermore, the above configuration is merely an example, and in the embodiment, known components other than the optical element 12 and the image sensor 11 can be appropriately combined and used.

[0032] <Reconstruction Processing Model> The reconstruction processing model 20 is a model (machine learning model) that takes an image as input and outputs an output image which is a hyperspectral image. For example, the reconstruction processing model 20 may use a model (network) in the form of a neural network (NN), such as a deep neural network (DNN). For example, the reconstruction processing model 20 may have a structure related to a convolutional neural network (CNN). For example, the reconstruction processing model 20 may have a structure related to transposed convolution (deconvolution).

[0033] The above is merely an example, and the network structure (internal structure) of the reconstruction processing model 20 can be any structure as long as it can convert the input image into the desired hyperspectral image. For example, the reconstruction processing model 20 may be a neural network corresponding to the mathematical model of reconstruction processing described below. Furthermore, although the following explanation uses the case where the hyperspectral image has 31 bands as an example, the number of bands in the hyperspectral image is not limited to 31 bands, but can be any number of bands, including tens or hundreds of bands.

[0034] The reconstruction processing model 20 performs a reconstruction process to reconstruct an image based on a matrix defined by the imaging process of the optical element 12 (e.g., observation matrix Φ) and an image (observation image) in which PSFs of each wavelength are convolved. For example, the reconstruction processing model 20 is a model that takes an image (e.g., compressed image CI) formed (imported) by the image sensor 11 as input and outputs a hyperspectral image generated by a reconstruction process on the input image.

[0035] For example, the reconstruction process can be formulated as a process that solves an optimization problem with respect to the observation matrix Φ defined by the optical system and the compressed image g, which is the acquired encoded image, as shown in equation (1) below.

[0036]

[0037] In equation (1), the first term f on the right-hand side represents the image that we originally want to reconstruct (e.g., a hyperspectral image). Here, since the number of data points in the observed image is significantly less than the number of data points in the image to be reconstructed (reconstructed image), there are infinitely many solutions that satisfy Φf - g = 0. However, by adding a normalization term as the second term, it may be easier to find the most plausible image as the reconstructed image (reconstructed image^f). Note that the second term (normalization term) does not have to be included in equation (1). Also, when "^f" is written for f, it is equivalent to "a symbol with "^" written directly above "f".

[0038] Regarding the normalization term, various types have been proposed for spectral images, and in this embodiment, any of these regularization terms can be applied. In the example of equation (1), R corresponds to the prior probability of the signal based on prior information (image resemblance), and it utilizes the sparsity that images generally possess, such as small differences between adjacent pixels. Note that τ is a balancing parameter. In this embodiment, a term called SSTV (Spatio-Spectral Total Variation) (Reference 1) is used as the normalization term, and it is optimized to minimize the differences between adjacent pixels in the spatial and wavelength dimensions during image reconstruction. Reference 1: Aggarwal, H. K., & Majumdar, A. (2016). Hyperspectral image denoising using spatio-spectral total variation. IEEE Geoscience and Remote Sensing Letters, 13(3), 442-446.

[0039] Thus, the reconstruction processing model 20 generates a hyperspectral image by performing a process that reconstructs the spatial and spectral information of the subject from the observed image, based on compressed sensing, for the observed image in which different convolution operations have been performed for each wavelength.

[0040] Optically encoded images, when the observation process is known (for example, the PSF of the optical element 12 and the wavelength sensitivity characteristics of the sensor (image sensor 11, etc.)), can have their real image information restored by performing appropriate signal processing using the reconstruction processing model 20. Therefore, the image processing system 1 performs signal processing using compressed sensing, a technique that reconstructs (restores) an object with high accuracy from a small amount of information by utilizing the sparsity of natural images. The image processing system 1 can perform different encoding for each wavelength component of the real image due to the wavelength-dependent PSF of the optical element 12. Therefore, the image processing system 1 can restore a hyperspectral image by performing image reconstruction processing based on compressed sensing using the reconstruction processing model 20.

[0041] <Inference Processing> Next, an example of the inference processing executed by the above-described image processing system 1 will be described. The image processing system 1 generates a hyperspectral image by inference processing using the reconstruction processing model 20.

[0042] First, the image processing system 1 acquires a captured image of the imaging target 2 irradiated with light and imaged by the imaging device 10 having the optical element 12 and the imaging element 11. For example, the image processing system 1 acquires a first image (image DI) in which the plurality of first structures 14a of the optical element 12 and the imaging element 11 image the subject. Further, the image processing system 1 acquires a compressed image CI generated from a second image in which the plurality of second structures 14b of the optical element 12 and the imaging element 11 image the subject.

[0043] The image processing system 1 controls the moving mechanism 15 to change the optical distance between the optical element 12 and the imaging element 11, and determines the position where the focus of the acquired first image DI is in focus. The image processing system 1 specifies the position (focus position) of the optical element 12 when the first image DI is in focus.

[0044] Then, the image processing system 1 inputs the captured image captured when the optical element 12 is at the specified focus position into the reconstruction processing model 20, and causes the reconstruction processing model 20 to output a hyperspectral image SI having a larger number of bands than the captured image. Thereby, the image processing system 1 generates a hyperspectral image from the captured image. For example, the imaging device 10 generates a compressed image CI from the second image captured when the optical element 12 is at the specified focus position. The image processing system 1 inputs the compressed image CI into the reconstruction processing model 20, and causes the reconstruction processing model 20 to output a hyperspectral image SI having a larger number of bands than the compressed image CI. Thereby, the image processing system 1 generates a hyperspectral image from the compressed image CI.

[0045] In this way, the image processing system 1 captures an image DI, which is a captured image (first image), using an auxiliary lens realized by a plurality of first structures 14a of the optical element 12 and the imaging element 11. Further, the image processing system 1 captures a captured image (second image) using a main lens realized by a plurality of second structures 14b of the optical element 12 and the imaging element 11, and generates a compressed image CI from the captured image (second image). The image processing system 1 uses the moving mechanism 15 to move the optical element 12, adjusts the optical distance between the optical element 12 and the imaging element 11, and performs autofocus control to search for the in-focus position of the image DI. The image processing system 1 uses the in-focus position of the image DI as the in-focus position of the second image. That is, the image processing system 1 generates the compressed image CI from the captured image (second image) captured using the main lens and the imaging element 11 realized by the second structure 14b at the specified in-focus position. The image processing system 1 generates a hyperspectral image from the compressed image CI. Thereby, the image processing system 1 enables autofocus control using the optical element 12 having a plurality of fine structures 14 in the imaging of the hyperspectral image.

[0046] <Example of device configuration> Next, the configuration of an information processing device 100, which is an example of a device (computer) that executes information processing according to the embodiment, will be described. FIG. 2 is a diagram showing a configuration example of a device that executes information processing according to the embodiment. For example, FIG. 2 is a diagram showing a configuration example of the information processing device 100. In FIG. 2, the description will be made with one information processing device 100, but there may be a plurality of information processing devices 100. In this case, each of the plurality of information processing devices 100 may have at least a part of the various configurations shown below, and communication may be performed between each of the plurality of information processing devices 100, and the processing may be executed in cooperation (jointly).

[0047] For example, in the image processing system 1, the imaging device 10 may have some of the functions of the information processing device 100, while the remaining functions may be held by other devices such as a server device (cloud server, etc.). In this case, the imaging device 10, which has some of the functions of the information processing device 100, and the server device, which has the remaining functions, may communicate and cooperate (jointly) perform processing. Alternatively, if the imaging device 10 does not have the functions of the information processing device 100, the information processing device 100, such as the server device, may communicate with the imaging device 10 and perform inference processing (reconstruction processing) using the captured image received from the imaging device 10.

[0048] From here, an overview of the information processing device 100 shown in Figure 2 will be described. As illustrated in Figure 2, the information processing device 100, which is an example of a device that performs information processing according to the embodiment, has a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 may also have an input unit that receives various operations from the administrator of the information processing device 100, a display unit for displaying information, and an output unit for outputting information as sound. Examples of the input unit include a keyboard and a mouse. Examples of the display unit include a liquid crystal display. Examples of the audio output unit include a speaker.

[0049] For example, the information processing device 100 may have a display as a display unit. The display unit, such as a display, has the function of displaying an image inferred (generated) by the inference process. The display unit, such as a display, displays an image such as a hyperspectral image generated by the inference process.

[0050] The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to a predetermined network, such as the Internet, by wired or wireless connection, and transmits and receives information with other information processing devices, such as terminal devices used by users receiving inference services provided by the information processing device 100.

[0051] The storage unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. As shown in Figure 2, the storage unit 120 according to this embodiment has a model information storage unit 121 and an inference result storage unit 122.

[0052] The model information storage unit 121 according to this embodiment stores information about a model. The model information storage unit 121 stores the reconstructed processing model 20.

[0053] For example, the model information storage unit 121 stores information (model data) that indicates the structure of the model (network). Figure 3 is a diagram showing an example of the model information storage unit 121 according to an embodiment. In the example shown in Figure 3, the model information storage unit 121 includes items such as "model ID," "purpose," and "model data."

[0054] "Model" refers to information used to identify the model, such as the model's name. Note that "Model" may also store identification information (model ID) to identify the model. "Purpose" indicates the purpose of the corresponding model. "Model Data" refers to the model's data. Figure 3 shows an example where conceptual information such as "MDT20" is stored in "Model Data," but in reality, it includes various information constituting the model, such as information about the network included in the model, functions, and model parameter information learned through the learning process.

[0055] In the example shown in Figure 3, the reconstruction processing model indicates that its purpose is "reconstruction processing." The reconstruction processing model indicates that it is a model used for reconstruction processing. Furthermore, the model data for the reconstruction processing model is indicated as model data MDT20. For example, the reconstruction processing model is reconstruction processing model 20.

[0056] The model information storage unit 121 is not limited to the above and may store various types of information depending on the purpose. For example, the model information storage unit 121 may store multiple reconstruction processing models. In this case, the model information storage unit 121 may store information indicating the reconstruction processing model (e.g., a model ID). For example, the model information storage unit 121 stores information that identifies each reconstruction processing model, associating it with that reconstruction processing model. For example, the model information storage unit 121 may store multiple reconstruction processing models optimized according to the loss function used in the learning process.

[0057] Furthermore, for example, when the information processing device 100 performs learning processing, the model information storage unit 121 may store a camera image generation simulator 21 (see Figure 8), which is an example of a compressed image generation model. In this way, the model information storage unit 121 may store a hyperspectral image generation model 200 that includes the camera image generation simulator 21 and the reconstruction processing model 20.

[0058] The inference result storage unit 122 according to this embodiment stores information related to the results of the inference process. The inference result storage unit 122 stores the generated hyperspectral image (reconstructed image). For example, the inference result storage unit 122 stores the hyperspectral image generated using the optical element 12 and the reconstruction processing model 20. For example, the storage unit 120 functions as an image data memory (also called "inference result data memory") that stores the hyperspectral image generated by the inference process.

[0059] The inference result storage unit 122 functions as an inference result data memory that stores images inferred (generated) by the inference process. For example, the inference result storage unit 122 stores hyperspectral image data of, for example, 256 × 256 × 31, i.e., a 31-band hyperspectral image, generated by the reconstruction processing model 20. However, the inference result storage unit 122 is not limited to the above and may store various types of information depending on the purpose.

[0060] Returning to Figure 2, let's continue the explanation. The control unit 130 is implemented by a processor, such as a CPU (Central Processing Unit), which executes a program stored inside the information processing device 100 using RAM or the like as a working area. The program is an information processing program, such as an inference processing program. The control unit 130 is also implemented by an integrated circuit, such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). As shown in Figure 2, the control unit 130 includes an acquisition unit 131, a focus control unit 132, an inference unit 133, and an output unit 134.

[0061] The acquisition unit 131 acquires information. The acquisition unit 131 acquires information from an external information processing device. The acquisition unit 131 acquires information from the user's terminal device. The acquisition unit 131 acquires information from the storage unit 120. The acquisition unit 131 acquires information from the model information storage unit 121 and the inference result storage unit 122.

[0062] The acquisition unit 131 acquires models and other data to be used in inference processing from the model information storage unit 121. For example, the acquisition unit 131 acquires the reconstruction processing model 20 from the model information storage unit 121.

[0063] The acquisition unit 131 acquires the captured images (first image and second image) captured by the optical element 12 and the image sensor 11. For example, the acquisition unit 131 acquires the first image captured when the image sensor 11 receives visible light that has arrived through a plurality of first structures 14a. Also, for example, the acquisition unit 131 acquires the second image captured when the image sensor 11 receives light including invisible light that has arrived through a plurality of second structures 14b. The acquisition unit 131 may also acquire a compressed image generated from the second image.

[0064] The focus control unit 132 adjusts the optical distance between the optical element 12 and the image sensor 11 based on the image captured by the image sensor 11. For example, the focus control unit 132 moves the optical element 12 by controlling the movement mechanism 15 and adjusts the optical distance between the optical element 12 and the image sensor 11 based on the first image. The focus control unit 132 and the movement mechanism 15 may be included in the imaging device 10. The movement mechanism 15 has an actuator such as a motor that moves at least one of the optical element 12 or the image sensor 11. The movement mechanism 15 may have a position sensor (not shown). For example, the movement mechanism 15 can detect the position of the optical element 12 while moving the optical element 12 in a direction facing the image sensor 11 using the actuator. The focus control unit 132 identifies the position of the optical element 12 when it determines that the first image is in focus as the in-focus position. The focus control unit 132 may move at least one of the optical element 12 or the image sensor 11 by controlling the movement mechanism 15. In this case, the focus control unit 132 identifies the position of at least one of the optical element 12 or the image sensor 11 at the time it determines that the first image is in focus as the focus position. In this way, the focus control unit 132 adjusts the optical distance between the optical element 12 and the image sensor 11 based on the first image to bring the first image into focus.

[0065] The inference unit 133 performs inference processing to infer information. The inference unit 133 performs inference processing based on information from an external information processing device and information stored in the storage unit 120. The inference unit 133 performs inference processing using information (model) stored in the model information storage unit 121.

[0066] The inference unit 133 performs inference processing based on the information acquired by the acquisition unit 131. The inference unit 133 performs inference processing using a machine learning model (model). The inference unit 133 generates a hyperspectral image by performing inference processing using the reconstruction processing model 20. The image sensor 11 captures a second image by receiving light, including invisible light, that has arrived through a plurality of second structures 14b at a specified focal position. The inference unit 133 inputs the captured image (compressed image CI) to the reconstruction processing model 20. The inference unit 133 then generates a hyperspectral image from the captured image by causing the reconstruction processing model 20 to output a hyperspectral image with a larger number of bands than the captured image. For example, if multiple models are registered in the model information storage unit 121, the inference unit 133 selects a model corresponding to the inference processing from among the multiple models registered in the model information storage unit 121 and performs inference processing using the selected model.

[0067] The output unit 134 performs output processing to output various types of information. The output unit 134 functions as a transmission unit that transmits various types of information. The output unit 134 performs output processing by transmitting information to an external information processing device. The output unit 134 transmits information to an external information processing device. For example, if another device (also called a "display device") has a display, the output unit 134 transmits various types of information to the display device. In this case, the output unit 134 transmits the hyperspectral image generated by the inference unit 133 to the display device which has a display. The display device displays the received hyperspectral image on its display.

[0068] If the imaging device 10 has a display, the output unit 134 transmits the generated hyperspectral image to the imaging device 10. In this case, the output unit 134 transmits the hyperspectral image generated by the inference unit 133 to the imaging device 10 which has a display. The imaging device 10 displays the received hyperspectral image on its display.

[0069] The output unit 134 may transmit the model stored in the model information storage unit 121 to another device. For example, the output unit 134 may transmit the reconstruction processing model 20 to another device that performs reconstruction processing (also called a "service provider device"). Multiple reconstruction processing models 20 may be registered in the model information storage unit 121. In this case, for example, the output unit 134 may transmit to the service provider device a reconstruction processing model 20 specified by the service provider device from among the multiple reconstruction processing models 20 registered in the model information storage unit 121.

[0070] Furthermore, the output unit 134 of the information processing device 100, which has a display, performs output processing by displaying information on the display. In this case, the output unit 134 displays the hyperspectral image generated by the inference unit 133 on the display.

[0071] <Example of a flowchart for an image processing method (focus control processing)> From here, an example of a flowchart for the focus control processing included in the image processing method according to the embodiment will be explained using Figures 4 to 6. Figure 4 is a flowchart of the focus control processing procedure. Figure 5 is a diagram showing an example of an image captured by the imaging device 10 that performs focus control processing. Figure 6 is a diagram for explaining the focus control processing procedure. The flowchart in Figure 4 is started, for example, when there is an input instructing the imaging device 10 to start operation. For example, the imaging device 10 may start the focus control processing when it acquires an image (first image).

[0072] In step S101 shown in Figure 4, the focus control unit 132 starts moving the optical element 12, which has a plurality of first structures 14a and a plurality of second structures 14b. For example, as shown in Figures 5 and 6, the focus control unit 132 controls the movement mechanism 15 to move the optical element 12 so that the optical distance L (focal length) between the optical element 12 and the image sensor 11 decreases in the order of L1, L2, and L3. The focus position is determined by the distance between the optical element 12 and the image sensor 11. The image sensor 11 may be fixed or movable. When the image sensor 11 is fixed, the focus position can be determined by the position of the optical element 12.

[0073] Figures 5 and 6(a) show that the distance L between the optical element 12 and the image sensor 11 is L1. Figure 6(b) shows that the optical element 12 has moved until the distance L is L2. Figure 6(c) shows that the optical element 12 has moved until the distance L is L3. As a result, the focus control unit 132 changes the optical distance between the optical element 12 and the image sensor 11. The direction of movement of the optical element 12 may be towards the image sensor 11 or away from it.

[0074] Step S101 is an example of a process in which at least one of an optical element having a plurality of fine structures arranged on the same plane, or an image sensor facing the optical element and receiving light that has arrived through the optical element, is moved in such a way as to change the optical distance between the optical element and the image sensor.

[0075] In step S102, the image sensor 11 captures a first image by receiving visible light that has arrived through the plurality of first structures 14a. As shown in Figure 6(a), when the distance L is L1, the imaging device 10 captures image DI1. As shown in Figure 6(b), when the optical element 12 moves until the distance L is L2, the imaging device 10 captures image DI2. As shown in Figure 6(c), when the optical element 12 moves until the distance L is L3, the imaging device 10 captures image DI3. Images DI1, DI2, and DI3 are examples of first images. In this way, as shown in Figures 5 and 6, images DI1, DI2, and DI3 are captured according to the position of the optical element 12. Images DI1, DI2, and DI3 may be color images or monochrome images.

[0076] The first response characteristics of the multiple first structures 14a are, for example, optical characteristics that guide visible light from the light incident on the multiple first structures 14a to the corresponding position on the image sensor 11. Therefore, since images DI1, DI2, and DI3 are color images or monochrome images that mimic the human eye, known autofocus techniques can be used.

[0077] Using this autofocus control, in step S103, the focus control unit 132 determines whether the first image is in focus. If the focus control unit 132 determines that the first image is not in focus, it returns to step S102 and repeats the processes in steps S102 and S103 until it determines that the first image is in focus. Here, the focus control unit 132 determines that image DI2 is in focus among images DI1, DI2, and DI3, and proceeds to step S104.

[0078] In step S104, the focus control unit 132 identifies the position of the optical element 12 that it has determined to be in focus on the first image as the focus position and terminates the process.

[0079] Step S104 is an example of a step to determine the position of at least one of the optical elements or the image sensor when it is determined that the first image captured by the image sensor receiving visible light that has arrived through a plurality of first structures has been brought into focus.

[0080] <Example of a flowchart for the image processing method (inference processing)> Next, the inference processing included in the image processing method according to the embodiment will be explained using Figures 5 to 7. Figure 7 is a flowchart of the inference processing procedure. The flowchart in Figure 7 starts, for example, when the focus position of the optical element 12 is determined. For example, the image processing system 1 starts inference processing for the second image at the focus position of the optical element 12. The image sensor 11 captures the second image by receiving light including invisible light that has arrived through a plurality of second structures 14b. The imaging device 10 (signal processing unit) generates a compressed image from the second image.

[0081] In step S201, the acquisition unit 131 acquires a compressed image generated from a second image captured by the image sensor 11 receiving light, including invisible light, that has arrived via a plurality of second structures 12b at the focal position of the specified optical element 12. For example, as shown in Figures 5 and 6(a) to (c), when the distance L between the optical element 12 and the image sensor 11 is L1, L2, and L3, the imaging device 10 generates compressed images CI1, CI2, and CI3 from the second images captured by the image sensor 11, respectively. Therefore, in step S201, the acquisition unit 131 acquires the compressed image CI2, which was generated from the second image captured at the focal position of the optical element 12, from among the generated compressed images CI1, CI2, and CI3.

[0082] Step S201 is an example of a step to acquire a compressed image of a second image captured when an image sensor receives light, including invisible light, that has arrived via a plurality of second structures, at at least one of the specified optical elements or image sensors.

[0083] Next, in step S202, the inference unit 133 inputs the compressed image CI to the reconstruction processing model and causes the reconstruction processing model to output a hyperspectral image with a greater number of bands than the compressed image CI, thereby generating a hyperspectral image from the compressed image CI. For example, the inference unit 133 inputs the compressed image CI2 generated from the second image captured at the focus position where the distance L between the optical element 12 and the image sensor 11 is L2 to the reconstruction processing model and causes the reconstruction processing model to output a hyperspectral image with a greater number of bands than the compressed image CI2, thereby generating a hyperspectral image from the compressed image CI2.

[0084] Step S202 is an example of a process for generating a hyperspectral image from a compressed image by inputting the compressed image into a reconstruction processing model that outputs an output image which is a hyperspectral image, and causing the reconstruction processing model to output a hyperspectral image with a larger number of bands than the compressed image.

[0085] <Effects> According to the imaging device 10 of this embodiment, the optical element 12 has a two-lens configuration consisting of a main lens realized by a plurality of second structures 14b and an auxiliary lens realized by a plurality of first structures 14a.

[0086] As a result, the imaging device 10 performs autofocus control using the first image captured by the auxiliary lens realized by the multiple first structures 14a. Since the first image is an image that captures visible light, the imaging device 10 can focus on the first image using known autofocus techniques. On the other hand, the second image captured by the main lens has significantly different characteristics from the image that captures visible light, so the imaging device 10 cannot focus on the second image using known autofocus techniques. Therefore, the imaging device 10 uses the first image to identify the in-focus position of the first image and uses the identified in-focus position as the in-focus position of the second image. The second image captured at the in-focus position of the first image is estimated to be in focus. Then, a compressed image CI from which a hyperspectral image can be reconstructed is generated from the second image which is estimated to be in focus. As a result, the imaging device 10 can indirectly perform autofocus control to focus on the second image captured by the main lens. As a result, sharp hyperspectral images can be acquired even in dynamic scenes, such as when the subject is moving or when the imaging device 10 is mounted on equipment such as a drone.

[0087] In the imaging device 10 according to the embodiment described above, the auxiliary lens is realized by a plurality of first structures 14a, and the main lens is realized by a plurality of second structures 14b. The plurality of first structures 14a and the plurality of second structures 14b are arranged on substantially the same plane. With this configuration, the imaging device 10 can manufacture both lenses simultaneously using the same process, and can assemble both lenses simultaneously. Therefore, the main lens may be a metasurface.

[0088] The auxiliary lens may be a lens for capturing images that can utilize the conventional autofocus method. That is, the auxiliary lens realized by a plurality of first structures 14a is an example of a first lens that receives visible light, and the first lens may be a conventional lens (concave or convex lens) or a metasurface. The main lens may be a lens for capturing images other than hyperspectral images that cannot utilize the conventional autofocus method. The main lens is an example of a second lens that receives invisible light, and the second lens may be a conventional lens (concave or convex lens). The first lens and the second lens are arranged on substantially the same plane. The focus control unit 132 may set the focal length between the optical element 12 and the image sensor 11 using the first lens, and set the focal length of the second lens based on the focal length.

[0089] The second lens may output an image to the image sensor 11 that is more difficult to detect contrast with compared to the first lens. The contrast detection method is based on the assumption that the contrast between adjacent pixels is maximized at the focal plane, and considers the location where the contrast is maximized to be the focal plane. On the other hand, there are lenses for which the above assumption does not hold. For example, if the PSF is spatially spread out, if the PSF has characteristics such as different values ​​for each wavelength, or if the PSF has multiple peaks and multiple images overlap, it may not be possible to expect the contrast to be maximized at the focal plane. In such cases, the second lens for which the above assumption does not hold will output an image to the image sensor 11 that is more difficult to detect contrast with compared to the first lens.

[0090] <Learning Process> Here, an example of the learning process for the reconstruction processing model 20 will be explained with reference to Figure 8. Figure 8 is a diagram illustrating the overview of the learning process. In the example shown below, we will explain the case in which a hyperspectral image generation model 200, which includes a camera image generation simulator 21 that functions as a simulator for an imaging device 10 having optical elements 12 and a reconstruction processing model 20, is learned.

[0091] For example, the hyperspectral image generation model 200 is a machine learning model (also simply called a "model") for updating the parameters corresponding to the optical element 12 and the parameters of the reconstruction processing model 20 for converting an image captured using the optical element 12 into a hyperspectral image.

[0092] The camera image generation simulator 21 is a compressed image generation model that generates a compressed image by compressing an input image, which is a hyperspectral image, based on parameters corresponding to an optical element 12 having a microstructure. For example, the camera image generation simulator 21 is an image generation simulator that infers (generates) images captured by a camera through simulation. For example, the camera image generation simulator 21 is a mathematical model for simulating images captured by an optical element 12 having a microstructure. The camera image generation simulator 21 is a model in a format that can be processed by a computer, and any model can be adopted as long as its parameters can be updated through learning processes.

[0093] The camera image generation simulator 21 generates image data with a reduced number of bands in response to input hyperspectral image data with several tens of bands or more. For example, the camera image generation simulator 21 generates image data with a reduced number of bands in response to input hyperspectral image data with 20 to 50 bands.

[0094] The camera image generation simulator 21 generates image data with a reduced number of bands in a 31-band hyperspectral image, based on the input of 256 × 256 × 31 hyperspectral image data. For example, the camera image generation simulator 21 generates a compressed image with the number of bands in a 31-band hyperspectral image reduced to 3 bands, based on the input of 256 × 256 × 31 hyperspectral image data. Note that the image generated by the camera image generation simulator 21 is not limited to a 3-band (RGB) color image; the image sensor 11 may also produce a 1-band (monochrome) image. For example, if the imaging device 10 captures a 256 × 256 × 1 image (monochrome image), the camera image generation simulator 21 may output a 1-band (monochrome) image.

[0095] From here, we will explain the overall outline of the learning process, using the hyperspectral image generation model 200 described above as an example. For example, the hyperspectral image generation model 200 is trained using a learning model stored in an image data memory (learning data memory) that stores hyperspectral images used in the learning process. The learning data memory functions as a memory unit that stores images used in the learning process. For example, the learning data memory stores hyperspectral image data of, for example, 256 × 256 × 31, i.e., 31-band hyperspectral images, which are the learning data.

[0096] The learning process uses an error calculator, which functions as a processing unit that performs error calculations. The error calculator calculates the root mean squared error (RMSE) between the image input to the hyperspectral image generation model 200 from the learning data memory and the image output from the hyperspectral image generation model 200 during the learning process. For example, the error calculator calculates the root mean squared error (RMSE) between the hyperspectral image input to the camera image generation simulator 21 and the hyperspectral image output from the reconstruction processing model 20. Note that the root mean squared error is just one example of an error, and the error is not limited to the root mean squared error; any arbitrary index such as mean squared error or mean absolute error may be used.

[0097] For example, the learning process is performed to train the camera image generation simulator 21 and the reconstruction processing model 20 to minimize the value of the loss function, which includes elements related to the error between the input image and the output image, such as the root mean square error from the error calculator, based on the hyperspectral image data which is the training data. In this way, the image processing system 1 performs optimization processing of the hyperspectral image generation model 200 by a learning process using a loss function that includes elements related to the error.

[0098] Next, we will explain the overview of the learning process shown in Figure 8. As shown in Figure 8, the camera image generation simulator 21 generates a compressed image g by compressing the input image f, which is a hyperspectral image, using the observation matrix Φ. For example, the camera image generation simulator 21 is a model that, when an input image f is input, derives a compressed image g (=Φf) by performing calculations using the input image f and the observation matrix Φ. In this way, the camera image generation simulator 21 is an image generation simulator that generates a compressed image, which is a simulation result of an image captured using the optical element 12, by converting it into an image with fewer bands than a hyperspectral image using parameters corresponding to the optical element 12.

[0099] Here, the observation matrix Φ is a parameter corresponding to the imaging device 10. That is, the observation matrix Φ is a parameter corresponding to the PSF of the optical element 12. Therefore, if the image sensor 11 of the imaging device 10 is fixed and the observation matrix Φ is determined, the PSF of the optical element 12 corresponding to that observation matrix Φ can be derived, and the physical configuration (structure 14, etc.) of the optical element 12 can be designed based on the derived PSF.

[0100] Furthermore, as shown in Figure 8, the reconstruction processing model 20 generates a reconstructed image ^f, which is a hyperspectral image, from the compressed image g. In this way, during the learning process, the reconstruction processing model 20 receives a compressed image generated by the camera image generation simulator 21 as input and outputs an output image, which is a hyperspectral image with a larger number of bands than the input compressed image.

[0101] The image processing system 1 performs a learning process on a hyperspectral image generation model 200, which includes a camera image generation simulator 21 and a reconstruction processing model 20. For example, the image processing system 1 performs an optimization process to update the hyperspectral image generation model 200, which includes parameters corresponding to the optical elements 12, by performing a learning process that learns the hyperspectral image generation model 200 using a loss function. For example, in the learning process, the hyperspectral images stored in the learning data memory are used as input images and ground truth information (ground truth images) for the hyperspectral image generation model 200 (camera image generation simulator 21).

[0102] The image processing system 1 performs an optimization process for the hyperspectral image generation model 200 by a learning process using a loss function that includes an element related to the error between the input image and the output image. Based on the hyperspectral images stored in the training data memory, the image processing system 1 performs a learning process (optimization process) to adjust the parameters of the camera image generation simulator 21 and the reconstruction processing model 20 so that the value of the loss function that includes an element related to the error between the input image and the output image is minimized.

[0103] For example, the image processing system 1 performs a learning process using methods such as backpropagation so that the reconstructed image ^f output by the hyperspectral image generation model 200, which has the input image f as input, approaches the input image f. Note that the learning method used by the image processing system 1 for the learning process of the hyperspectral image generation model 200 can be any learning method as long as the hyperspectral image generation model 200 is capable of learning, and a detailed explanation is omitted.

[0104] The above is merely an example, and the image processing system 1 may perform learning processing on the hyperspectral image generation model 200, etc., using various methods. For example, the image processing system 1 may use a hyperspectral image encoded by the encoding member 30 as an input image to perform the learning process. Furthermore, although the above example described the case in which learning processing is performed on the hyperspectral image generation model 200 including the camera image generation simulator 21, the learning process may be performed on only the reconstruction processing model 20.

[0105] In this case, the learning process is performed by taking a compressed image as input and using training data, which contains the corresponding hyperspectral image as the ground truth, to adjust the parameters of only the reconstruction model 20. As mentioned above, the learning process may be performed by the image processing system 1 or by another system.

[0106] Furthermore, the element relating to the error between the input image and the output image is merely one example of the elements included in the loss function, and the image processing system 1 may use a loss function that includes various other elements in addition to the element mentioned above to perform the training process of the hyperspectral image generation model 200.

[0107] For example, the loss function may include an element relating to the maximum value of the response characteristics of the optical element 12 for each wavelength. For example, the loss function may include an element relating to the maximum value of the PSF for each wavelength of the optical element 12. In this case, the image processing system 1 performs an optimization process of the hyperspectral image generation model 200 by a learning process using a loss function that includes an element relating to the maximum value of the PSF. This allows the image processing system 1 to add a limit to the maximum PSF size to the loss function, thereby reducing the computational cost during inference by reducing the PSF (corresponding to the convolution kernel) size.

[0108] Furthermore, the loss function may include elements related to the homogenization of the response characteristics of the optical element 12 for each wavelength. For example, the loss function may include elements related to the homogenization of the PSF of the optical element 12 for each wavelength. In this case, the image processing system 1 performs an optimization process of the hyperspectral image generation model 200 by a learning process using a loss function that includes elements related to the homogenization of the PSF of the optical element 12 for each wavelength. As a result, the image processing system 1 can add a loss function that homogenizes the PSF size across wavelengths, thereby homogenizing the spatial resolution.

[0109] Furthermore, the loss function may include elements relating to the response characteristics of the optical element 12 according to its field of view. For example, the loss function may include elements relating to the PSF for each field of view of the optical element 12. In this case, the image processing system 1 performs an optimization process of the hyperspectral image generation model 200 by a learning process using a loss function that includes elements relating to the PSF for each field of view of the optical element 12.

[0110] The loss function may include an element relating to the transmittance of the optical element 12. For example, the loss function may include an element relating to the total transmittance of the optical element 12. In this case, the image processing system 1 performs an optimization process of the hyperspectral image generation model 200 by a learning process using a loss function that includes an element relating to the total transmittance of the optical element 12. This allows the image processing system 1 to add the total lens transmittance to the loss function, thereby improving efficiency, preventing stray light, and so on.

[0111] The image processing system 1 performs the training process of the hyperspectral image generation model 200 using a loss function that includes any combination of the elements described above. For example, the image processing system 1 performs the training process of the hyperspectral image generation model 200 using a loss function that includes at least an element related to error. For example, the image processing system 1 performs the training process of the hyperspectral image generation model 200 using a loss function that includes multiple elements.

[0112] Then, an optical element 12 having response characteristics such as PSF corresponding to the observation matrix Φ of the camera image generation simulator 21 after the learning process is created. An optical element 12 having PSF corresponding to the observation matrix Φ of the camera image generation simulator 21 after the learning process is created. In this way, the optical element 12 is designed based on the parameters updated by the optimization process. The optical element 12 is shaped based on the response characteristics such as PSF converted from the parameters updated by the optimization process. Then, the optical element 12 created based on the PSF corresponding to the observation matrix Φ of the camera image generation simulator 21 after the learning process is mounted on the imaging device 10.

[0113] As described above, when the image processing system 1 performs learning processing, the information processing device 100 may also perform learning processing. In this case, the storage unit 120 of the information processing device 100 may have a learning data storage unit that stores data used for learning processing. For example, the learning data storage unit of the storage unit 120 stores data used for processing. For example, the learning data storage unit stores data used for learning (also called "learning data"). For example, the learning data storage unit functions as an image data memory (also called "learning data memory") that stores hyperspectral images used for learning processing. For example, the learning data storage unit includes items such as "data ID" and "hyperspectral image".

[0114] "Data ID" indicates identification information for identifying each training data. "Hyperspectral image" indicates a hyperspectral image used as training data identified by the data ID. The hyperspectral images stored in the training data storage unit are used in the training process as input images and correct information (correct images) for the hyperspectral image generation model 200 (camera image generation simulator 21). For example, the hyperspectral image may be a hyperspectral image captured by a line-scan type hyperspectral camera or the like.

[0115] Furthermore, the learning data storage unit is not limited to the above and may store various types of information depending on the purpose. For example, the learning data storage unit may store information to be inferred. In this case, the learning data storage unit may have an item indicating whether the information is information used for learning (learning data) or information to be inferred (inference target data), and may store information indicating the type, such as "for learning" or "for inference," as information corresponding to that item.

[0116] Furthermore, for example, when the information processing device 100 performs a learning process, the storage unit 120 may have a loss function information storage unit that stores information about the loss function used during the learning process. The loss function information storage unit of the storage unit 120 stores various information about the loss function. The loss function information storage unit stores the functions used in the loss function. For example, the loss function information storage unit stores the functions included in the loss function. For example, the loss function information storage unit stores functions related to errors such as the least squares error.

[0117] The loss function information storage unit is not limited to the above and may store various types of information depending on the purpose. The loss function information storage unit may also store information indicating which of the functions will be used as the loss function during the learning process. For example, the loss function information storage unit may store a flag indicating which of the functions will be used as the loss function during the learning process. When a function whose element (evaluation target) is the error between the input image and the output image, and a function whose element (evaluation target) is the maximum value of the response characteristics of the optical element 12 for each wavelength, are used as the loss function during the learning process, the loss function information storage unit stores a flag associated with these functions to indicate that they will be used as the loss function.

[0118] Furthermore, when the information processing device 100 performs learning processing, the control unit 130 of the information processing device 100 has a learning unit. In this case, the control unit 130 functions as the error calculator described above.

[0119] For example, the acquisition unit 131 acquires training data to be used for training the model from the training data storage unit of the storage unit 120. The acquisition unit 131 acquires a hyperspectral image generation model 200 which includes a camera image generation simulator 21 that generates a compressed image in which the input image, which is a hyperspectral image, is compressed based on parameters corresponding to the optical element 12 having a fine structure 14, and a reconstruction processing model 20 that takes an image as input and outputs an output image, which is a hyperspectral image. The acquisition unit 131 acquires a loss function to be used when training the hyperspectral image generation model 200.

[0120] The learning unit of the control unit 130 performs a learning process to learn a model (machine learning model). The learning unit performs an optimization process to update the hyperspectral image generation model 200, which includes parameters corresponding to the optical elements 12, by performing a learning process that learns the hyperspectral image generation model 200 using a loss function. The learning unit performs an optimization process of the hyperspectral image generation model 200 by performing a learning process that uses a loss function related to errors such as least squares error.

[0121] The learning unit performs learning processing based on information from an external information processing device and information stored in the storage unit 120. The learning unit performs learning processing based on learning data stored in the learning data storage unit. The learning unit stores the model generated by the learning processing in the model information storage unit 121.

[0122] The learning unit performs learning processing based on the information acquired by the acquisition unit 131. The learning unit learns the model using various machine learning techniques. For example, the learning unit learns the parameters of the model. The learning unit learns the model using various machine learning techniques. For example, the learning unit learns the parameters of the hyperspectral image generation model 200.

[0123] The learning unit performs learning processing using the learning data (training data) stored in the learning data storage unit. For example, the learning unit learns a hyperspectral image generation model 200 by performing learning processing using the learning data. The learning unit learns a hyperspectral image generation model 200 that includes a camera image generation simulator 21 that generates a compressed image in which the input image, which is a hyperspectral image, is compressed based on parameters corresponding to the optical element 12 having a microstructure, and a reconstruction processing model 20 that takes an image as input and outputs an output image, which is a hyperspectral image.

[0124] The learning unit performs learning processing so that when a hyperspectral image is input, the information output by the hyperspectral image generation model 200 approaches the input hyperspectral image (ground truth information). For example, when using training data, the learning unit trains the hyperspectral image generation model 200 so that when a hyperspectral image is input, it outputs the input hyperspectral image (ground truth information). In this case, the learning unit performs learning processing using methods such as backpropagation so that the image output by the hyperspectral image generation model 200 when a hyperspectral image is input approaches the hyperspectral image.

[0125] The learning method used by the learning unit is not limited to the above; any learning method can be employed as long as it can learn a desired model such as the hyperspectral image generation model 200. Furthermore, the network structure (internal structure) of the model such as the hyperspectral image generation model 200 can be any structure as long as it can output the desired information in response to the input. For example, the model such as the hyperspectral image generation model 200 may be a neural network (NN) such as a deep neural network (DNN). In addition, any type of model (function) can be used for the model such as the hyperspectral image generation model 200 as long as it can output the desired information.

[0126] Furthermore, a device other than the information processing device 100 may perform part of the learning process. For example, a device other than the information processing device 100 (a machine learning device) may perform the learning process, and a foundation model may be generated by pre-training performed by the machine learning device. In this case, the information processing device 100 may receive the model (foundation model) that has been pre-trained by the machine learning device from the machine learning device, and the learning unit may generate a model such as a hyperspectral image generation model 200 by fine-tuning the foundation model.

[0127] Furthermore, if the information processing device 100 does not perform learning processing, the information processing device 100 does not need to have a learning unit. That is, models such as the hyperspectral image generation model 200 may be learned (generated) by a device other than the information processing device 100 (a machine learning device). For example, if a machine learning device learns a model such as the hyperspectral image generation model 200, the information processing device 100 may acquire (receive) the model such as the hyperspectral image generation model 200 from the machine learning device and perform processing such as inference processing using that model. For example, the information processing device 100 may perform inference processing using the reconstruction processing model 20 acquired from the machine learning device.

[0128] <Program> A program can also be created that describes the processing performed by the computer (information processing device 100, etc.) included in the image processing system 1, etc., according to the above embodiment, in a language that can be executed by a computer. In one embodiment, the information processing device 100 can be implemented by installing an information processing program that performs the above information processing as package software or online software on a desired computer. For example, by having the information processing device execute the above information processing program, the information processing device can be made to function as the information processing device 100. In addition, the category of information processing device includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), and slate terminals such as PDA (Personal Digital Assistant). Furthermore, the functions of the information processing device 100 may be implemented on a cloud server.

[0129] Figure 9 shows an example of a computer that executes an information processing program. Computer 1000 includes, for example, memory 1010, CPU 1020, hard disk drive interface 1030, disk drive interface 1040, serial port interface 1050, video adapter 1060, and network interface 1070. These components are connected by a bus 1080.

[0130] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1031. The disk drive interface 1040 is connected to the disk drive 1041. A removable storage medium, such as a magnetic disk or optical disk, is inserted into the disk drive 1041. A serial port interface 1050 is connected to, for example, a mouse 1051 and a keyboard 1052. A video adapter 1060 is connected to, for example, a display 1061.

[0131] Here, the hard disk drive 1031 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. The information described in the above embodiment is stored, for example, in the hard disk drive 1031 or the memory 1010.

[0132] Furthermore, the information processing program is stored in the hard disk drive 1031 as a program module 1093 containing instructions to be executed by the computer 1000, for example. Specifically, the program module 1093 containing instructions for each process executed by the information processing device 100 described in the above embodiment is stored in the hard disk drive 1031.

[0133] Furthermore, the data used for information processing by the information processing program is stored as program data 1094, for example, in the hard disk drive 1031. The CPU 1020 then reads the program module 1093 and program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as needed and executes the procedures described above.

[0134] Furthermore, the program module 1093 and program data 1094 related to the information processing program are not limited to being stored in the hard disk drive 1031; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1041 or the like. Alternatively, the program module 1093 and program data 1094 related to the information processing program may be stored in another computer connected via a network such as a LAN (Local Area Network) or WAN (Wide Area Network) and read by the CPU 1020 via a network interface 1070.

[0135] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in this embodiment. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are included in the scope of the present invention.

[0136] 1 Image processing system 10 Imaging device (camera) 11 Image sensor 12 Optical element 13 Transparent substrate 14 Structure 14a First structure 14b Second structure 15 Moving mechanism 20 Reconstruction processing model 21 Camera image generation simulator (compressed image generation model) 100 Information processing device 110 Communication unit 120 Storage unit 121 Model information storage unit 122 Inference result storage unit 130 Control unit 131 Acquisition unit 132 Focus control unit 133 Inference unit 134 Output unit

Claims

1. An imaging device comprising an optical element including a first lens for receiving visible light and a second lens for receiving non-visible light, and an image sensor for receiving light that has reached through the optical element, wherein the first lens and the second lens are arranged substantially on the same plane.

2. The imaging apparatus according to claim 1, further comprising a focus control unit that sets the focal length between the optical element and the image sensor using the first lens, and sets the focal length of the second lens based on the focal length.

3. The imaging apparatus according to claim 1, wherein the second lens causes the image sensor to output an image in which contrast is less easily detected compared to the first lens.

4. The imaging device according to claim 2, wherein the optical element and the image sensor are arranged facing each other, and the imaging device has a moving mechanism for moving at least one of the optical element or the image sensor so as to change the optical distance between the optical element and the image sensor.

5. The imaging apparatus according to claim 4, wherein the image sensor captures a first image by receiving visible light that has reached through the first lens, and the focus control unit controls the movement mechanism to adjust the optical distance between the optical element and the image sensor based on the first image.

6. The imaging apparatus according to claim 5, wherein the focus control unit determines that the first image is in focus and identifies the position of at least one of the optical element or the image sensor; the image sensor captures a second image by receiving invisible light that has reached the identified position through the second lens; and the imaging apparatus has an inference unit that inputs a compressed image generated from the second image to a reconstruction processing model that outputs an output image which is a hyperspectral image, and causes the reconstruction processing model to output a hyperspectral image with a greater number of bands than the compressed image, thereby generating a hyperspectral image from the compressed image.

Images