Image information generation program, hyperspectral camera, and artificial satellite

Abstract

[Problem] To provide an image information generation program, a hyperspectral camera, and an artificial satellite for generating hyperspectral data even when a linear variable bandpass filter is used. [Solution] This hyperspectral camera 1 comprises a linear variable band-pass filter 13, an image sensor 12 in which the linear variable band-pass filter 13 is installed on an imaging surface, a lens 14, and a control unit 10, wherein the control unit 10 functions as: a first conversion means 101 that performs position matching between one image and an image that is temporally adjacent to said image, for a plurality of images in which an object to be photographed is scanned and photographed at predetermined time intervals in a direction in which the transmission wavelength of the linear variable band-pass filter 13 changes, obtains a first conversion operator for matching, and converts the temporally adjacent images and a wavelength table; and an image combination means 102 that combines a plurality of images that have been converted on the basis of the converted wavelength table to obtain hyperspectral data 113.

Description

Image information generation program, hyperspectral camera and artificial satellite

[0001] The present invention relates to an image information generation program, a hyperspectral camera, and an artificial satellite.

[0002] As a conventional technique, an image information generation program that stitches images captured in a plurality of courses for an image captured by a hyperspectral camera has been proposed (see, for example, Non-Patent Document 1).

[0003] The image information generation program disclosed in Patent Document 1 uses a hyperspectral camera to perform imaging in the east-west direction for color correction in addition to multiple imaging operations in the north-south direction, and constructs a color correction model by comparing the color information (observed values) of overlapping fields between the east-west direction captured images and the north-south direction captured images.

[0004] Japanese Unexamined Patent Application Publication No. 2008-258679

[0005] On the other hand, the aforementioned hyperspectral camera uses a line sensor capable of measuring a plurality of spectra, and requires a spectroscope such as a slit, so there is a limit to the miniaturization of the hyperspectral camera. Accordingly, miniaturization of a hyperspectral camera has been proposed by omitting the spectroscope using a linear variable bandpass filter.

[0006] However, in the aforementioned image information generation program, imaging is performed using a line sensor capable of measuring multiple spectra as a hyperspectral camera, so the captured information is hyperspectral data, and an overall image can be obtained by stitching the obtained hyperspectral data. When a hyperspectral camera configured with a linear variable bandpass filter is used, the captured information is image data having different transmission wavelengths in the one-dimensional direction, so there is a problem that hyperspectral data cannot be obtained simply by stitching the captured data as it is. In addition, the captured image includes distortion caused by the lens of the hyperspectral camera, parallax depending on the imaging position, and blur caused by instability of the imaging posture, so there is a problem that accurate hyperspectral data cannot be obtained simply by adding the images together.

[0007] Therefore, the object of the present invention is to provide an image information generation program, a hyperspectral camera, and a satellite that can generate hyperspectral data even when imaging is performed using a hyperspectral camera with a linear variable bandpass filter.

[0008] One aspect of the present invention provides the following image information generation program, hyperspectral camera, and artificial satellite to achieve the above objective.

[0009] [1] An image information generation program that causes a computer to function as a first transformation means that scans and captures a plurality of images taken with a hyperspectral camera consisting of an image sensor and a lens on which a linear variable bandpass filter is installed on the imaging surface, at predetermined time intervals in the direction in which the transmission wavelength of the linear variable bandpass filter changes, matches the position of one of the plurality of images with an image adjacent to it in time, determines a first transformation operator for matching the image adjacent to it in time, and transforms the image adjacent to it in time and a wavelength table that associates the coordinates and transmission wavelengths of the image adjacent to it using the first transformation operator, and synthesizes the plurality of images transformed based on the wavelength table transformed by the first transformation means to obtain hyperspectral data. [2] The image information generation program according to [1], which further functions as a second transformation means that matches the position of one image among a plurality of images of the hyperspectral data with an image adjacent in time, based on the resolution of the image sensor and / or the focal length of the lens, determines a second transformation operator for matching the image adjacent in time to the first image, and transforms the image adjacent in time using the second transformation operator to obtain hyperspectral data. [3] A hyperspectral camera comprising a linear variable bandpass filter, an image sensor and lens on which the linear variable bandpass filter is installed on the imaging surface, and a control unit, wherein the control unit functions as a first transformation means that scans and captures a plurality of images of an object at predetermined time intervals in the direction in which the transmission wavelength of the linear variable bandpass filter changes, matches the position of one of the plurality of images with an image adjacent to it in time, determines a first transformation operator for matching the image from the image adjacent to it, and transforms the image adjacent to it and a wavelength table that associates the coordinates and transmission wavelengths of the image adjacent to it using the first transformation operator, and an image synthesis means that synthesizes the plurality of images transformed based on the wavelength table transformed by the first transformation means to obtain hyperspectral data.[4] The hyperspectral camera according to [3], wherein the control unit further functions as a second transformation means for matching the position of one image among a plurality of images of the hyperspectral data with an image adjacent in time, based on the resolution of the image sensor and / or the focal length of the lens, determining a second transformation operator for matching the image from the image adjacent in time to the first image, and transforming the image adjacent in time using the second transformation operator to obtain hyperspectral data. [5] An artificial satellite equipped with the hyperspectral camera according to [3] or [4], wherein the imaging target is the Earth and the orbit is such that the transmission wavelength of the linear variable bandpass filter changes. [6] The image information generation program according to [2], wherein the first transformation means is a transformation means using an affine transformation matrix and the second transformation means is a transformation means using a homography transformation matrix. [7] The hyperspectral camera according to [4], wherein the first transformation means is a transformation means using an affine transformation matrix and the second transformation means is a transformation means using a homography transformation matrix.

[0010] According to the inventions of claims 1, 3, and 5, hyperspectral data can be generated even when imaging is performed using a hyperspectral camera with a linear variable bandpass filter. According to the inventions of claims 2 and 4, hyperspectral data can be generated that suppresses distortion due to the lens of the hyperspectral camera and parallax due to the imaging position. According to the inventions of claims 6 and 7, the first transformation means can be a transformation means using an affine transformation matrix, and the second transformation means can be a transformation means using a homography transformation matrix.

[0011] Figure 1 is a schematic diagram showing an example of the configuration of a satellite equipped with a hyperspectral camera according to an embodiment. Figure 2 is a schematic diagram showing the configuration of the camera. Figure 3 is a block diagram showing the configuration of the camera. Figure 4A is a schematic diagram showing the pixel positions of the image sensor and the corresponding pixel positions and transmission wavelengths of the bandpass filter. Figure 4B is a schematic diagram showing the pixel positions of the image sensor and the corresponding pixel positions and transmission wavelengths of the bandpass filter. Figure 5 is a schematic diagram for explaining the imaging operation. Figure 6A is a schematic diagram for explaining the relationship between the wavelength table and the captured image group and the matching operation between images. Figure 6B is a schematic diagram for explaining the relationship between the wavelength table and the captured image group and the matching operation between images. Figure 7A is a schematic diagram for explaining the configuration of the wavelength table and images obtained by the first conversion operation and the hyperspectral data obtained therefrom. Figure 7B is a schematic diagram for explaining the configuration of the wavelength table and images obtained by the first conversion operation and the hyperspectral data obtained therefrom. Figure 8 is a flowchart for explaining the first conversion operation. Figure 9 is a flowchart for explaining the second conversion operation. Figure 10A is a schematic diagram illustrating the shooting range determined by the satellite's position relative to Earth and the characteristics of its camera, as well as the resulting distortion of the wavelength table. Figure 10B is a schematic diagram illustrating the shooting range determined by the satellite's position relative to Earth and the characteristics of its camera, as well as the resulting distortion of the wavelength table.

[0012] [Embodiment] (Satellite Configuration) Figure 1 is a schematic diagram showing an example of the configuration of a satellite equipped with a hyperspectral camera according to the embodiment.

[0013] This artificial satellite 2 includes a camera 1, a battery and solar panel that share power to drive the camera 1, a communication unit for communicating with a ground base station (not shown), an attitude control unit for controlling the flight attitude, and components such as a CPU (Central Processing Unit) that control each of these units.

[0014] Artificial satellite 2 flies in Earth's orbit, and the subject of camera 1 is the Earth's surface. The area to be photographed, A, is traced in the direction of travel, D. The orbital altitude is, for example, 680 km.

[0015] (Camera Configuration) Figure 2 is a schematic diagram showing the configuration of camera 1.

[0016] Camera 1 includes a control unit 10 for processing information, a storage unit 11 for storing information, an image sensor 12 which is an image sensor for capturing incoming light, a bandpass filter 13 provided on the imaging surface of the image sensor 12 that transmits light of a specific wavelength depending on the position, and a lens 14 for forming an image.

[0017] The control unit 10 consists of a CPU and the like, and controls each part and executes various programs.

[0018] The memory unit 11 is composed of a storage medium such as flash memory and stores information.

[0019] The image sensor 12 is a CMOS image sensor or the like, with 1.3 million pixels, a size of 6.8 mm x 5.4 mm, and a resolution of 450 m / pixel.

[0020] The bandpass filter 13 has the same size as the image sensor 12, and its transmitted wavelength changes, for example, from 400 to 800 nm, depending on the direction of propagation D (Figure 1).

[0021] Lens 14 is a fixed-focus lens with a focal length of 8 mm. When combined with the image sensor 12, the angle of view is 46° vertically and 38° horizontally, and the area captured is 460 km vertically and 576 km horizontally.

[0022] (Configuration of the information processing device) Figure 3 is a block diagram showing the configuration of camera 1.

[0023] The control unit 10 functions as an image receiving means 100, a first conversion means 101, an image synthesis means 102, a second conversion means 103, etc., by executing the image information generation program 110, which will be described later.

[0024] The image receiving means 100 acquires the image output by the image sensor 12 at time intervals of 0.5 seconds and stores it in the storage unit 11 as image information 111.

[0025] The first transformation means 101 estimates the amount of movement between multiple images contained in the image information 111, generates a transformation operator to align each image, and performs a transformation on the image information 111 and the wavelength table 112 which shows the correspondence between the transmission wavelength of the bandpass filter 13 and the coordinates on the image. The transformation operator is a parallel projection transformation, and as an example, it is an affine transformation matrix.

[0026] The image synthesis means 102 synthesizes the wavelength components of the image information 111 converted by the first conversion means 101 to generate hyperspectral data, and stores it in the storage unit 11 as hyperspectral data 113.

[0027] The second conversion means 103 estimates the correspondence between multiple images contained in the hyperspectral data 113, generates a conversion operator to align each image, performs the conversion on the hyperspectral data 113, and stores the corrected hyperspectral data 113 in the storage unit 11. The conversion operator mainly performs projection operations to correct image distortion due to lens distortion and parallax, and one example is a homography transformation matrix.

[0028] The storage unit 11 stores an image information generation program 110, image information 111, a wavelength table 112, hyperspectral data 113, etc., which cause the control unit 10 to operate as the means 100-103 described above.

[0029] (Operation of the Information Processing Device) Next, the operation of this embodiment will be explained by dividing it into (1) shooting operation, (2) first conversion operation, and (3) second conversion operation.

[0030] (1) Shooting Operation First, the artificial satellite 2 flies in Earth's orbit, and the camera 1 takes pictures of the ground shooting range A in the direction of travel D every 0.5 seconds, and stores multiple images output from the image sensor 12 in the storage unit 11 as image information 111. Note that a wavelength table 112 showing the pixel positions of the image information 111 and the corresponding transmission wavelengths is stored in the storage unit 11 in advance.

[0031] Figures 4A and 4B are schematic diagrams showing the pixel positions of the image sensor 12 and the corresponding pixel positions and transmission wavelengths of the bandpass filter 13, respectively.

[0032] As shown in Figure 4A, the image sensor 12 has pixel units arranged in the x and y directions. Similarly, as shown in Figure 4B, the wavelength table 112 of the bandpass filter 13 has coordinates corresponding to the pixel units of the image sensor 12, with a wavelength set for each coordinate. The wavelength in the y direction of the bandpass filter 13 varies, for example, from 400 to 800 nm. This wavelength is set as the z axis in the hyperspectral data.

[0033] Figure 5 is a schematic diagram illustrating the shooting operation.

[0034] Image information 111 includes multiple images 111t0 to 111t6… obtained by taking images of the ground area A over time in the direction of travel D, and each image 111t0 to 111t6… has different wavelength components in the spectral direction z. Image 102a is obtained by combining images 111t0 to 111t6…, and since the image in the square frame portion of image 102a contains all wavelength components, hyperspectral data 113 can be obtained by describing the wavelength components in the z direction.

[0035] Ideally, hyperspectral data 113 should be obtainable from the above images 111t0 to 111t6... However, due to distortion caused by the lens 14 during shooting, parallax at the shooting position (including the effect of the Earth being a sphere), and instability of the shooting posture, the images 111t0 to 111t6... will not overlap as they are. Therefore, before combining the images 111t0 to 111t6..., a first transformation described below is performed.

[0036] (2) First conversion operation FIG. 8 is a flowchart for explaining the first conversion operation. FIGS. 6A and 6B are schematic diagrams for explaining the relationship between a wavelength table and a captured image group and the matching operation between images.

[0037] First, as shown in FIG. 6A, the first conversion means 101 reads N images (S10) and reads the wavelength table λ (S11).

[0038] Next, the first conversion means 101 performs image enhancement processing on the N images (S12). Since the enhancement processing is a preparatory operation for facilitating the extraction of feature points described below, it may be omitted depending on the image content.

[0039] Next, the first conversion means 101 extracts coordinates of a plurality of feature points of images 112n and 112n+1 (S13).

[0040] Next, as shown in FIG. 6B, the first conversion means 101 performs matching on the extracted feature points of images 112n and 112n+1, and selects the top three sets of coordinate positions (m11, m12, m13) (m21, m22, m23) with high matching degrees (S14).

[0041] Next, the first conversion means 101 obtains an affine transformation matrix An+1 from (m11, m12, m13) (m21, m22, m23) (S15).

[0042] FIGS. 7A and 7B are schematic diagrams for explaining the configuration of a wavelength table and an image obtained by the first conversion operation, and hyperspectral data obtained from the foregoing.

[0043] Next, the first conversion means 101 executes affine transformation (A=AnAn+1)*1 on the image 112n+1 and the wavelength table λ, to obtain an image 112sn+1 and a wavelength table λsn+1 as shown in FIG. 7A (S16)

[0044] Next, the image composition means 102 stores the image 112sn+1 in hyperspectral data HSD(x, y, z)*1 based on the wavelength table λsn+1 (S17).

[0045] The above steps S13 to S14 are repeated from n=1 to n=N (S18). As shown in FIG. 7B, the image combining means 102 combines each wavelength component of the image information 111 converted by the first converting means 101 to obtain hyperspectral data 113.

[0046] (3) Second Conversion Operation FIG. 9 is a flowchart for explaining the second conversion operation.

[0047] First, the second converting means 103 reads Z spectral images HSD(z), HSD(z+1), ... from the hyperspectral data 113 (S20).

[0048] Next, let HSD(z)s = HSD(z). Although HSD(z)s is a result of homography transformation, the reference data read in the first step is data before homography transformation (S21).

[0049] Next, the second converting means 103 performs image enhancement processing on the spectral images HSD(z) and HSD(z+1) (S22). Since this enhancement processing is a preparatory operation for facilitating the extraction of feature points described below, it may be omitted depending on the image content.

[0050] Next, the second converting means 103 extracts coordinates of a plurality of feature points of the spectral images HSD(z) and HSD(z+1) (S23).

[0051] Next, the second converting means 103 performs matching on the extracted feature points of the spectral images HSD(z) and HSD(z+1), and extracts four or more sets of coordinates (S24).

[0052] Next, the second converting means 103 obtains a homography transformation matrix Mz+1 from four or more sets of coordinates (S25).

[0053] Next, the second converting means 103 performs homography transformation (M=MzMz+1)*1 on the spectral image HSD(z+1) to obtain a spectral image HSD(z+1)s (S26)

[0054] The above steps S22 to S26 are repeated from z=1 to z=Z (S27), and hyperspectral data 113 subjected to the second conversion is obtained.

[0055] (Effects of the Embodiment) According to the embodiment described above, a wavelength table 112 corresponding to the linearly variable bandpass filter 13 is prepared, the wavelength table 112 is transformed by an affine transformation matrix corresponding to the positional shift between multiple images included in the image information 111, hyperspectral data 113 is generated using the coordinates of the transformed wavelength table, and each hyperspectral data is further changed by a mohography transformation matrix corresponding to the positional shift between the hyperspectral data included in the hyperspectral data 113 to obtain a sampled hyperspectral data 113. As a result, even when shooting is performed using a hyperspectral camera with a linearly variable bandpass filter, the effects of distortion by the lens 14 during shooting, parallax at the shooting position (including the effect of the Earth being a sphere), and instability of the shooting posture can be suppressed, and hyperspectral data can be generated.

[0056] [Other Embodiments] The present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention.

[0057] Figures 10A and 10B are schematic diagrams illustrating the imaging range determined by the satellite's position relative to the Earth and the characteristics of the camera, as well as the resulting distortion of the wavelength table.

[0058] In the specifications of camera 1 in the above embodiment, if the radius R of the Earth is 6378 km, the altitude of the orbit of artificial satellite 2 is 680 km. Therefore, the parallax between the altitude of the image center and the edge of the image is approximately 44 km, resulting in significant distortion. The affine transformation by the first transformation means 101 alone would result in a large discrepancy. For this reason, homography transformation by the second transformation means 103 is necessary. On the other hand, if the resolution of camera 1 is increased, for example, if the focal length of lens 14 is extended or the size of image sensor 12 is reduced due to increased resolution, the parallax becomes small, approximately 20 m, and the effect of distortion is reduced. In this case, homography transformation can be omitted.

[0059] However, as the resolution increases, the image becomes more susceptible to the influence of the attitude control accuracy of satellite 2. To achieve an overlap of 80% or more in consecutive images, the required accuracy is approximately 4° for camera 1 (field of view 38°) and 0.1° for a field of view of 1°. Here, while the error in attitude control has little effect on distortion, it has a large effect on movement and rotation, so there is a high possibility that images can be superimposed using only affine transformation, and in this case, homography transformation can be omitted.

[0060] In the above embodiment, the functions of each means 100 to 103 of the control unit 10 were implemented by program, but all or part of each means may be implemented by hardware such as ASIC. Furthermore, the program used in the above embodiment can be stored and provided on a recording medium such as a CD-ROM. Also, the steps described in the above embodiment can be rearranged, deleted, or added without altering the essence of the present invention. Potential for industrial use

[0061] This invention provides an image information generation program, a hyperspectral camera, and a satellite that generate hyperspectral data even when imaging is performed using a hyperspectral camera with a linear variable bandpass filter.

[0062] 1: Camera 2: Satellite 10: Control unit 11: Memory unit 12: Image sensor 13: Bandpass filter 14: Lens 100: Image receiving means 101: First conversion means 102: Image synthesis means 103: Second conversion means 110: Image information generation program 111: Image information 112: Wavelength table 113: Hyperspectral data

Claims

1. An image information generation program that causes a computer to function as an image information generation program that uses a hyperspectral camera consisting of an image sensor and a lens with a linear variable bandpass filter installed on the imaging surface, to scan and capture a target object at predetermined time intervals in the direction in which the transmission wavelength of the linear variable bandpass filter changes, to perform a positional matching between one of the multiple images and an image adjacent to it in time, to determine a first transformation operator for matching the image from the image adjacent to it, to transform the image adjacent to it in time using the first transformation operator, and to transform the image adjacent to it and a wavelength table that associates the coordinates and transmission wavelengths of the image adjacent to it in time using the first transformation operator, and to function as an image synthesis means that synthesizes the multiple images transformed based on the wavelength table transformed by the first transformation means to obtain hyperspectral data.

2. The image information generation program according to claim 1, further functioning as a second transformation means that matches the position of one image among a plurality of images of the hyperspectral data with an image adjacent in time, based on the resolution of the image sensor and / or the focal length of the lens, determines a second transformation operator for matching the image adjacent in time to the first image, and transforms the image adjacent in time using the second transformation operator to obtain hyperspectral data.

3. A hyperspectral camera comprising a linear variable bandpass filter, an image sensor and lens on which the linear variable bandpass filter is installed on the imaging surface, and a control unit, wherein the control unit functions as a first transformation means that scans and captures a plurality of images of an object at predetermined time intervals in the direction in which the transmission wavelength of the linear variable bandpass filter changes, matches the position of one of the plurality of images with an image adjacent to it in time, determines a first transformation operator for matching the image from the image adjacent to it, and transforms the image adjacent to it and a wavelength table that associates the coordinates and transmission wavelengths of the image adjacent to it using the first transformation operator, and an image synthesis means that synthesizes the plurality of images transformed based on the wavelength table transformed by the first transformation means to obtain hyperspectral data.

4. The hyperspectral camera according to claim 3, wherein the control unit further functions as a second transformation means for matching the positions of one image among a plurality of images of the hyperspectral data with temporally adjacent images based on the resolution of the image sensor and / or the focal length of the lens, determining a second transformation operator for matching the temporally adjacent images to the one image, and transforming the temporally adjacent images using the second transformation operator to obtain hyperspectral data.

5. An artificial satellite equipped with the hyperspectral camera described in claim 3 or 4, wherein the imaging target is the Earth, and the orbit is such that the transmission wavelength of the linear variable bandpass filter changes.

6. The image information generation program according to claim 2, wherein the first transformation means is a transformation means using an affine transformation matrix, and the second transformation means is a transformation means using a homography transformation matrix.

7. The hyperspectral camera according to claim 4, wherein the first transformation means is a transformation means using an affine transformation matrix, and the second transformation means is a transformation means using a homography transformation matrix.

Images