Estimation device

Abstract

An estimation device 1 for estimating the extent of river plume comprises: a model modification unit (133) that acquires a trained SSM which was pre-trained using image data pairs (data pairs for pre-training), and constructs a first SSM in which the trained SSM is changed such that a final layer is composed of the two classes of river plume and non-river plume; a model training unit (135) that acquires other image data pairs (data pairs for transfer learning) consisting of a plurality of satellite images and data in which tags indicating river plume and non-river plume are added to regions obtained by dividing each satellite image into prescribed unit regions, and constructs a second SSM for estimating the extent of river plume by inputting the other image data pairs to the first SSM and performing machine learning; and an inference unit (14) that inputs a target satellite image for estimation to the second SSM and estimates the extent of the river plume included in the target satellite image for estimation.

Description

Estimation device

[0001] The present disclosure relates to an estimation device for estimating the river plume range.

[0002] Substances such as suspended solids and nutrients flowing from land through rivers into the ocean greatly contribute to coastal biological production (such as seaweed farming and algal growth). In order to analyze the impact of human activities on the marine ecosystem, it is necessary to accurately grasp the range of substance inflow from rivers to the ocean (hereinafter referred to as the "river plume range").

[0003] A method of obtaining the river plume range using satellite data (e.g., images) acquired by artificial satellites near the estuary has been adopted. Non-Patent Document 1 discloses that by converting the brightness value of each band of satellite data into the concentration of suspended particulate matter (SPM) and setting a predetermined threshold value for the SPM concentration, the boundary of the river plume range is determined.

[0004] Ocean Color Remote Sensing of Suspended Sediments along a Continuum from Rivers to River Plumes: Concentration, Transport, Fluxes and Dynamics. [Searched on September 25, 2024], Internet <URL: https: / / www.mdpi.com / 2072-4292 / 14 / 9 / 202>

[0005] However, when converting the brightness value of each band of satellite data into the SPM concentration, the operator needs to perform an operation of setting a predetermined coefficient. In Non-Patent Document 1, the above coefficient is set using observation data near the estuary. Also, the SPM concentration for determining the boundary of the river plume range requires visual judgment by the observer. For this reason, there is a problem that a lot of labor by hand is required.

[0006] The present disclosure has been made in view of the above circumstances, and an object thereof is to provide an estimation device capable of estimating the river plume range with simple processing.

[0007] An estimation device according to one aspect of the present disclosure is an estimation device for estimating the range of a river plume, comprising: a model modification unit that acquires a pre-trained Semantic Segmentation Model (SSM) using a pair of image data, and constructs a first SSM by modifying the pre-trained SSM so that the final layer consists of two classes: river plumes and non-river plumes; a model learning unit that acquires other pairs of image data, each consisting of a plurality of satellite images and data in which tags indicating river plumes and non-river plumes are attached to each region obtained by dividing each satellite image into a predetermined unit region, inputs the other pairs of image data into the first SSM for machine learning, and constructs a second SSM for estimating the range of a river plume; and an inference unit that inputs a satellite image to be estimated into the second SSM and estimates the range of a river plume included in the satellite image to be estimated.

[0008] According to this disclosure, it becomes possible to estimate the extent of a river plume using a simple process.

[0009] Figure 1 is a block diagram showing the configuration of the estimation device and its peripheral equipment according to the embodiment. Figure 2 is a flowchart showing the pre-training process performed by the estimation device according to the embodiment. Figure 3 is a flowchart showing the transfer learning process performed by the estimation device according to the embodiment. Figure 4 is a flowchart showing the process for estimating the river plume range from satellite images. Figure 5 is a block diagram showing the hardware configuration of this embodiment.

[0010] The embodiments will be described below with reference to the drawings. Figure 1 is a block diagram showing the configuration of the estimation device 1 and its peripheral equipment according to an embodiment.

[0011] The estimation device 1 according to this embodiment constructs a semantic segmentation model for estimating the river plume range near the river mouth, reducing manual processing such as acquiring various observation data and making visual judgments by monitors in the area near the river mouth where the river plume range is to be estimated. A "semantic segmentation model" refers to a model that learns semantic segmentation images in which tags (annotations) are added to each region of an image that has been divided into predetermined unit regions. Hereinafter, the semantic segmentation model will be abbreviated as "SSM". The unit region may be, for example, one pixel of the image.

[0012] Estimation device 1 constructs an SSM (Second SSM, described later) capable of distinguishing between river plumes and non-river plumes, and uses this SSM to estimate the extent of the river plume. This will be explained in detail below.

[0013] As shown in Figure 1, the estimation device 1 is connected to the satellite image database 3, the data pair database 4, and the map display 5. The estimation device 1 may also be connected to the model database 2.

[0014] Model DB2 may store pre-trained SSMs. A pre-trained SSM is an SSM that has been machine-trained by inputting publicly available image data pairs from a network or other source into a basic training SSM. A pre-trained SSM refers to a model suitable for recognizing general images (e.g., satellite images) and performing machine learning. An "image data pair" refers to a combination of an image with tags (annotations) attached to each pixel and a satellite image.

[0015] Satellite image DB3 stores satellite images of the river mouth area acquired from artificial satellites. The satellite images are, for example, multiband optical images. In this embodiment, an example is described in which optical satellite images with only three RGB bands are used as the satellite images. Alternatively, optical satellite images with four or more bands may be used. By slightly modifying the structure of the SSM to match the attributes of the satellite images, it is possible to accommodate changes in the number of bands.

[0016] The DB4 data pair stores image data pairs that combine images tagged pixel by pixel with satellite images.

[0017] The map display unit 5 displays an image on the map showing the estimated results of the river plume range estimated by the estimation device 1.

[0018] The estimation device 1 comprises an acquisition unit 11, a storage unit 12, a calculation unit 13, an inference unit 14, and an analysis unit 15.

[0019] The acquisition unit 11 acquires a number of image data pairs (combinations of tagged images and satellite images) from the data pair DB 4 that are necessary for pre-training (pre-machine learning) the training SSM in the calculation unit 13 (for example, several hundred pairs). In other words, the acquisition unit 11 acquires general image data pairs necessary for pre-training. The acquisition unit 11 stores these image data pairs in the storage unit 12. Hereinafter, these image data pairs will be referred to as "pre-training data pairs". If the acquisition unit 11 can acquire a publicly available trained SSM from the model DB 2, it does not use the above pre-training data pairs, but instead acquires the trained SSM from the model DB 2 and stores it in the storage unit 12.

[0020] The acquisition unit 11 acquires satellite images from the satellite image database 3, for example, satellite images of the vicinity of a river mouth (for example, optical satellite images in RGB three bands), and stores them in the storage unit 12. Furthermore, the acquisition unit 11 acquires the number of satellite images from the satellite image database 3 necessary for machine learning the SSM (Surface Summarizer Model, described later) for river plume range estimation in the calculation unit 13. The acquisition unit 11 stores the acquired satellite images in the storage unit 12.

[0021] The storage unit 12 can be configured as a storage device such as a hard disk. The storage unit 12 stores the trained SSM acquired by the acquisition unit 11, the pre-training data pair, the trained SSM constructed by the calculation unit 13 (described later), and the inference results from the inference unit 14.

[0022] The calculation unit 13 includes an SSM construction unit 131, a pre-training unit 132, a model modification unit 133, a data pair generation unit 134, and a model learning unit 135.

[0023] The SSM construction unit 131 constructs the aforementioned training SSM. As mentioned above, an SSM is a model for learning semantic segmentation images in which tags are attached to each pixel (unit region) that makes up an image.

[0024] The pre-training unit 132 inputs the pre-training data pairs acquired by the acquisition unit 11 into the training SSM and performs machine learning to construct a trained SSM. That is, the pre-training unit 132 constructs a trained SSM capable of recognizing general images by pre-training the training SSM using the pre-training data pairs. By constructing a trained SSM, the number of image data pairs required when performing machine learning related to river plume range estimation in subsequent processing can be reduced. The pre-training unit 132 stores the constructed trained SSM in the storage unit 12.

[0025] On the other hand, if a pre-trained SSM is publicly available in the model DB2 and the acquisition unit 11 can acquire this pre-trained SSM, the pre-trained SSM acquired from the model DB2 is saved in the storage unit 12. In this case, pre-training by the pre-training unit 132 can be omitted.

[0026] In other words, if the acquisition unit 11 can directly acquire a trained SSM from the model DB 2, it saves this trained SSM as is to the storage unit 12. On the other hand, if the acquisition unit 11 cannot acquire a trained SSM from the model DB 2, the SSM construction unit 131 constructs a training SSM.

[0027] The model modification unit 133 reads the trained SSM from the storage unit 12 and performs a process to change this trained SSM into an SSM for river plume range estimation (hereinafter referred to as the "first SSM"). Here, it performs a process such as replacing the number of classes estimated in the final layer of the model with two classes: "river plume" and "non-river plume". The model modification unit 133 saves the first SSM to the storage unit 12. That is, the model modification unit 133 obtains a trained SSM obtained by pre-training a training SSM using a general image data pair (pre-training data pair), and constructs a first SSM by modifying the trained SSM so that the final layer has two classes: river plume and non-river plume. In other words, the first SSM has two classes in its final layer: river plume and non-river plume.

[0028] The data pair generation unit 134 reads the required number of satellite images (satellite images acquired from the satellite image DB 3) from the storage unit 12. Based on the operator's input, the data generation unit 134 identifies and tags the river plume ranges within the satellite images by visual inspection, etc., and generates tagged transfer learning data pairs that identify the river plume ranges. A "transfer learning data pair" is an example of another image data pair. In other words, the data pair generation unit 134 accepts input operations from the operator. When the operator observes the satellite images and performs an operation to distinguish between river plumes and non-river plumes in the satellite images, the data pair generation unit 134 generates transfer learning data pairs, which are pairs of tagged images and satellite images. It is also possible to perform the tagging operation automatically by image processing, etc. In this case, input operations by the operator are not required.

[0029] The model learning unit 135 inputs the above-mentioned transfer learning data pair into the first SSM and performs machine learning to construct an SSM for river plume range estimation (hereinafter referred to as the "second SSM"). The second SSM is a model capable of inferring river plumes and non-river plumes. The model learning unit 135 inputs the transfer learning data pair into the first SSM modified by the model modification unit 133 and performs machine learning to construct the second SSM for inferring river plumes and non-river plumes. That is, the model learning unit 135 acquires other image data pairs, each consisting of multiple satellite images and data in which tags indicating river plumes and non-river plumes are attached to each region obtained by dividing each satellite image into a predetermined unit region (for example, 1 pixel). The other image data pairs are input into the first SSM and machine learning is performed to construct the second SSM for river plume range estimation.

[0030] When the inference unit 14 performs the river plume range estimation process, it acquires the second SSM stored in the storage unit 12 and the satellite image to be estimated from the satellite image DB 3. The inference unit 14 inputs the satellite image to be estimated into the second SSM and performs inference of the river plume range in this satellite image. The inference unit 14 stores the inference result in the storage unit 12. In other words, the inference unit 14 inputs the satellite image to be estimated into the second SSM and estimates the river plume range contained in the satellite image to be estimated.

[0031] The analysis unit 15 reads the inference results for the river plume range stored in the storage unit 12. Using the read inference results, the analysis unit 15 performs analyses such as the time-series changes in the river plume range and the identification of sea areas that are statistically susceptible to the influence of river plumes, and displays the results on a display (not shown) or the like. The analysis unit 15 outputs the analysis results to the map display 5 as needed.

[0032] Next, the operation of the estimation device 1 according to this embodiment, configured as described above, will be explained. First, the pre-training process for constructing a trained SSM will be explained with reference to the flowchart shown in Figure 2. In step S11 of Figure 2, the acquisition unit 11 determines whether or not a trained SSM can be acquired in the model DB2. If a trained SSM can be acquired (S11; YES), the process proceeds to step S14; otherwise (S11; NO), the process proceeds to step S12.

[0033] In step S12, the SSM construction unit 131 constructs a training SSM, which is the basic SSM. As mentioned above, an SSM is a model that learns semantic segmentation images in which each pixel (unit region) of an image is tagged (annotated).

[0034] In step S13, the pre-training unit 132 reads the pre-training data pairs stored in the storage unit 12, inputs these pre-training data pairs into the training SSM, and performs machine learning on the training SSM. As a result, a trained SSM can be constructed.

[0035] In step S14, the pre-training unit 132 saves the trained SSM to the storage unit 12. Also, if a trained SSM is obtained from the model DB2, the obtained trained SSM is saved to the storage unit 12.

[0036] Next, referring to the flowchart shown in Figure 3, the transfer learning process for constructing a second SSM from a trained SSM will be explained. First, in step S31, the model modification unit 133 changes the trained SSM into a first SSM for river plume range estimation. Specifically, it performs processing such as replacing the number of classes estimated in the final layer of the trained SSM with two classes: "river plume" and "non-river plume".

[0037] In step S32, the data pair generation unit 134 generates a data pair for transfer learning (another image data pair) tagged with an identification of the river plume range.

[0038] In step S33, the model learning unit 135 inputs the transfer learning data pair into the first SSM and performs machine learning on the first SSM. As a result, a second SSM for river plume range estimation is constructed.

[0039] In step S34, the model learning unit 135 saves the second SSM to the storage unit 12. By performing transfer learning in this way, a second SSM capable of inferring river plumes and non-river plumes can be constructed.

[0040] Next, the procedure for estimating the river plume range will be explained with reference to the flowchart shown in Figure 4. First, in step S51, the inference unit 14 reads the second SSM stored in the storage unit 12.

[0041] In step S52, the inference unit 14 acquires the satellite image to be inferred from the storage unit 12.

[0042] In step S53, the inference unit 14 inputs the satellite image to be inferred to the second SSM and performs inference of the river plume range.

[0043] In step S54, the inference unit 14 displays the inference result on a display or the like. It also outputs it to an external map display 5. By viewing this inference result image, the operator can recognize the extent of the river plume included in the satellite image.

[0044] Thus, the estimation device 1 according to this embodiment is an estimation device 1 for estimating a river plume range, which acquires a learned SSM (Semantic Segmentation Model) pre-trained using an image data pair (pre-training data pair), and constructs a first SSM in which the learned SSM is modified such that the final layer has two classes: river plume and non-river plume. A model modification unit 133, a plurality of satellite images, and other image data pairs (transfer learning data pairs) formed by pairs with data in which tags indicating river plume and non-river plume are added to each region obtained by dividing each satellite image into predetermined unit regions are obtained. The other image data pairs are input into the first SSM for machine learning to construct a second SSM for estimating the river plume range, and an inference unit 14 that inputs a satellite image to be estimated into the second SSM and estimates the river plume range included in the satellite image to be estimated.

[0045] In this embodiment, it is possible to recognize the river plume range almost in real time using satellite images acquired daily. Also, since manual operation is not required or can be significantly reduced, it is possible to estimate the estuary plume range in a river with simple processing based on satellite images provided by artificial satellites.

[0046] In this embodiment, since the pre-training unit 132 inputs the pre-training image data pair into the learning SSM for machine learning to construct a learned SSM, a learned SSM can be constructed with a simple operation even when the learned SSM cannot be obtained from the model DB 2.

[0047] In this embodiment, since the model modification unit 133 constructs a first SSM in which the final layer has two classes: river plume and non-river plume, a first SSM capable of estimating the river plume range with high accuracy can be constructed.

[0048] In this embodiment, since the unit region to which the tag is assigned is one pixel of the satellite image, it is possible to estimate the river plume range with high accuracy.

[0049] As shown in Figure 5, the estimation device 1 of this embodiment described above can be a general-purpose computer system comprising, for example, a CPU (Central Processing Unit, processor) 901, memory 902, storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), communication device 904, input device 905, and output device 906. The memory 902 and storage 903 are storage devices. In this computer system, each function of the estimation device 1 is realized when the CPU 901 executes a predetermined program loaded onto the memory 902.

[0050] The estimation device 1 may be implemented on a single computer, or on multiple computers. Furthermore, the estimation device 1 may be a virtual machine implemented on a computer.

[0051] The program for the estimation device 1 can be stored on a computer-readable recording medium such as an HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), or DVD (Digital Versatile Disc), or it can be distributed via a network. A computer-readable recording medium is, for example, a non-transitory recording medium.

[0052] This disclosure is not limited to the embodiments described above, and numerous modifications are possible within the scope of its essence.

[0053] 1 Estimation device 2 Model DB 3 Satellite image DB 4 Data pair DB 5 Map display 11 Acquisition unit 12 Storage unit 13 Calculation unit 14 Inference unit 15 Analysis unit 131 SSM construction unit 132 Pre-training unit 133 Model modification unit 134 Data pair generation unit 135 Model learning unit

Claims

1. An estimation device for estimating the range of a river plume, comprising: a model modification unit that acquires a pre-trained SSM (Semantic Segmentation Model) using a pair of image data, and constructs a first SSM by modifying the pre-trained SSM so that the final layer consists of two classes: river plumes and non-river plumes; a model learning unit that acquires other pairs of image data, each consisting of multiple satellite images and data in which tags indicating river plumes and non-river plumes are attached to each region obtained by dividing each satellite image into a predetermined unit region, and inputs the other pairs of image data into the first SSM for machine learning to construct a second SSM for estimating the range of a river plume; and an inference unit that inputs a satellite image to be estimated into the second SSM and estimates the range of a river plume included in the satellite image to be estimated.

2. The estimation device according to claim 1, further comprising: an acquisition unit for acquiring the image data pair; and a pre-training unit for performing machine learning using the image data pair acquired by the acquisition unit to construct the trained SSM.

3. The estimation device according to claim 1 or 2, wherein the first SSM has two classes of final layers: river plume and non-river plume.

4. The estimation device according to claim 1 or 2, wherein the unit region includes at least one pixel of the satellite image.

Images