Vegetation determination system, method, program, and trained model

Abstract

[Problem] To provide a vegetation determination system that estimates vegetation proportions of a plurality of plant classifications on the basis of a satellite image. [Solution] In the present invention, an information processing system generates a second trained model by using a training data set which includes a satellite image as input data and includes, as teacher data, an output result of a first trained model generated using a UAV image. The information processing system comprises: a vegetation proportion map generation unit that generates the first trained model by performing training processing for determining a vegetation classification, by using a high-resolution vegetation classification training data set in which the UAV image and the teacher data indicating the vegetation classification are associated with each other, and generates a vegetation proportion map on the basis of a vegetation classification image outputted from the first trained model; a low-resolution vegetation proportion training data generation unit that generates a low-resolution vegetation proportion training data set in which the vegetation proportion map is associated, as the teacher data, with the satellite image; and a second training processing unit that performs machine learning processing for determining vegetation proportions from the satellite image, by using the low-resolution vegetation proportion training data set.

Description

Vegetation discrimination system, method, program, and trained model

[0001] The present invention relates to a system for discriminating vegetation using satellite images, a learning data generation system, a learning processing system, and methods, programs, methods, and trained models related to vegetation discrimination, learning data generation, and learning processing. In particular, it relates to a system for discriminating vegetation from satellite images using the estimation results of a machine learning model generated based on images taken by a drone or UAV as teacher data, a learning data generation system, a learning processing system, and methods, programs, methods, and trained models related to vegetation discrimination, learning data generation, and learning processing.

[0002] Conventionally, as techniques for discriminating the vegetation distribution of a target area, there are roughly three types of techniques: techniques using UAVs or drones (Techniques A-1, A-2), techniques using images taken by satellites (Techniques B-1, B-2, B-3), and techniques using both images taken by UAVs or drones and satellites (Technique C) (see Fig. 1).

[0003] As a technique using images taken by a UAV or drone, as a vegetation discrimination technique using high-resolution images taken from a relatively low altitude of a target area, it segments (an aggregate of polygons) by the SLIC method from the texture information of the Green band, and learns the teacher data, which is the position information of each grass species, by a random forest, and discriminates the grass species from the RGB image (Technique A-1 in Fig. 1, Kawamura et al. (2020) "Identification of upland rice crops and weeds from UAV images by the SLIC-RF algorithm", etc.).

[0004] Also, as in Patent Document 1, "Vegetation state detection method", a multi-spectral camera mounted on a UAV takes images of a plurality of bands of infrared and red from above a farm, field, orchard, or mountain covered with a protective net or disaster prevention net, integrates the taken image group, and maps the NDVI (Normalized Difference Vegetation Index) on the image for analysis (Technique A-2 in Fig. 1).

[0005] On the other hand, as a technology that utilizes images taken by satellite, there is a known technique for estimating pasture biomass based on the fact that there is a relationship between pasture biomass and the normalized difference water index (NDWI) (Technology B-1 in Figure 1, Makino, Tsukasa (2009), Utilization Technology of Remote Sensing and GIS in the Field of Grassland and Forage Crops, Hokuno 76:364-368, and Makino, Tsukasa (2019), Development and Dissemination of Productivity Evaluation and Management Technology for Grassland and Forage Crops Using Remote Sensing and GIS, Hokkaido Journal of Animal Science and Grassland Science 7:1-6, etc.).

[0006] Furthermore, as disclosed in Patent Document 2, "Method for Estimating Tree Species Using Satellite Images," a technique is disclosed in which the Earth's surface is photographed by a satellite, trees are automatically extracted from the image features, the extracted trees are compared with training data, a tree species estimation model is created using a multivariate analysis model, and the tree species of the extracted trees is estimated (Technology B-2 in Figure 1).

[0007] Furthermore, there is a technique called mixel decomposition, which considers a single pixel in a satellite image as a mixture of multiple pure spectral information (end-member spectra) of end-members (such as soil and plants), and performs vegetation classification by decomposing the mixed spectrum and inversely calculating the occupancy rate of each end-member (Technology B-3 in Figure 1, Chein-I Chang (2016), Linear Spectral Mixture Analysis. In: Real-time progressive hyperspectral image processing, pp37-73., etc.).

[0008] Furthermore, as a technology that utilizes both images taken by UAVs or drones and images taken by artificial satellites, there is the technology described in Patent Document 3 (Technology C in Figure 1). According to the technology in Patent Document 3, paragraph 0005 states that the macro measurement unit is a large device such as a hyperspectral camera and is difficult to mount on a drone, and paragraph 0011 states that the macro measurement unit is mounted on an artificial satellite. Paragraph 0012 also states that the micro measurement unit is intended for use on drones and the like. In addition, Figure 16 and subsequent figures in Patent Document 3 describe displaying the analysis results (macro analysis results) based on images taken by an artificial satellite, and then overlaying the analysis results based on images taken by a drone or the like.

[0009] Japanese Patent Publication No. 2022-087038, Japanese Patent Publication No. 2019-144607, Japanese Patent Publication No. 2021-012432

[0010] Current dairy and livestock farming operations are facing challenging business conditions due to the increasing environmental problems caused by the discharge of livestock manure and other waste products as the scale of operations expands, as well as the stagnation of self-sufficiency in feed and the impact of rising feed prices. Therefore, there is a need for proper treatment and utilization of livestock manure, improvement of feed quality, and improvement of feed self-sufficiency. Furthermore, using compost derived from livestock manure as fertilizer for growing various plants such as pasture grasses in grasslands can be expected to mitigate environmental problems caused by the discharge of livestock manure and other waste products, as well as reduce the cost of purchasing fertilizer.

[0011] On the other hand, since the amount of fertilizer required varies depending on the type of grass or other plant, it is necessary to accurately identify the types of grass and other plants. If supplemental fertilization is not performed according to the type of grass or other plant, or if less fertilizer is applied than necessary, the grass will be easily overtaken by weeds, and the proportion of grass will decrease significantly. This can lead to the need for immediate pasture renewal, a decrease in grass yield, and ultimately a decline in milk production, which in turn worsens dairy farm productivity.

[0012] Furthermore, when using compost derived from livestock manure for fertilization, excessive application of manure exceeding the standard application amount increases the K (potassium) and NO3-N (nitrate nitrogen) content of pasture grass, which can contribute to metabolic disorders in livestock. Therefore, it is important to accurately identify the types of pasture grass and other plants being used.

[0013] Thus, it is necessary to accurately understand the condition of the grasslands where various plants, such as pasture grasses that serve as raw materials for livestock feed, are grown. More specifically, by accurately determining the vegetation distribution, such as the proportion of leguminous grasses and weeds, it becomes possible to accurately determine the target yield and the amount of fertilizer to apply to the grasslands according to the type of pasture grass, such as leguminous grasses.

[0014] Furthermore, in order to improve the quality of self-sufficient feed, mowing management, grassland renewal, and grassland maintenance and improvement are important according to the vegetation. To do this, it is necessary to understand the extent of invasion by rhizome-type grass weeds and other species, as well as the factors causing the decline of sown pasture grasses. By understanding the condition of grasslands and factors of decline on a wide scale, such as the proportion of leguminous pasture grasses and the extent of invasion by grass weeds, it becomes possible to carry out "grassland management at the individual farm level," "planning of grassland management at the agricultural cooperative / contractor organization level," or "efficient grassland maintenance projects at the local government level."

[0015] Furthermore, the average grassland area per dairy farm in Hokkaido is vast, exceeding 50 hectares, and a significant proportion of these areas are remote, meaning that the grasslands managed cover a wide area. Moreover, the condition of the grasslands is greatly influenced not only by natural location but also by management conditions, so conditions are not necessarily similar even if the locations are close together, and it is not possible to infer the condition of neighboring fields from the condition of one field. For this reason, it is difficult to grasp the vegetation conditions of grasslands on a wide area and accurately through ground surveys.

[0016] Therefore, remote sensing and GIS technologies are expected to be useful as a way to grasp the state of grassland vegetation over a wide area and efficiently. Known technologies for vegetation discrimination using remote sensing include technologies that utilize images taken by drones or UAVs (Technologies A-1 and A-2 in Figure 1), technologies that utilize images taken by satellites (Technologies B-1, B-2, and B-3 in Figure 1), and technologies that utilize images taken by drones and satellites (Technology C in Figure 1).

[0017] First, the technology shown in Figure 1, Technology A-1, has the advantage of being able to distinguish from high-resolution (1 cm / pixel) RGB images, and can even identify small grass species communities of a few centimeters in size. However, because it uses high-resolution images, performing vegetation discrimination at high resolution limits the processing power of the computer, making it difficult to use in large grasslands (5 ha or more).

[0018] Furthermore, reducing the resolution resulted in pixel sizes larger than the size of individual grass colonies (several centimeters to several meters) in the grassland, leading to a significant decrease in discrimination accuracy. Additionally, the system was not robust enough to withstand changes in light conditions due to sunlight and weather, requiring training data for the same target area under different light conditions.

[0019] Furthermore, the technology shown in Figure 1, Technology A-2 (Patent Document 1), requires time-consuming annotation to handle high-resolution UAV or drone images, and because it uses UAVs or drones, it is difficult to target vast areas on a prefectural level, resulting in the disadvantage of being limited to specific, narrow areas.

[0020] Furthermore, while the technique B-1 shown in Figure 1 has the advantage of enabling wide-area imaging and easy data acquisition, it also has the disadvantage of being prone to mismatches between image resolution and plant size.

[0021] Furthermore, the system performed a process to output the classification result for each pixel, meaning that each pixel had to be classified into some category. More specifically, for example, if one pixel in a satellite image corresponds to 10m (since that entire pixel is not occupied by a single type of plant), there was a problem in that it could not accurately represent the plant distribution of that pixel. There was also a problem in that it could not handle grass species with small colony sizes (such as dandelions, Kentucky bluegrass, and sweet vernal grass).

[0022] Furthermore, the technique shown in Figure 1, Technique B-2, relies solely on satellite imagery, which limits its ability to identify tree species that spread over a relatively large area. This makes it difficult to apply to the vegetation distribution of pastures where various plants are intertwined.

[0023] Furthermore, technique B-3 (mixel decomposition) in Figure 1 is a method for calculating the proportion of each end member within a single pixel by assuming that the observed reflection spectrum of one pixel of satellite image is a linear combination of the area ratios of the reflection spectra of the individual components, given that the reflection spectral characteristics of each end member (soil, plants) that constitute the Earth's surface are known. However, in fields where multiple grass species are mixed, it is not easy to obtain the reflection spectra of individual end members (grass species), making it difficult to apply this method to applications such as classifying leguminous or grassy pastures from satellite images.

[0024] Furthermore, while the technology described in Technology C (Patent Document 3) utilizes both images taken by drones and images taken by satellites, in areas where high-resolution data from drones is unavailable, the analysis results from low-resolution images taken by satellites are not used to supplement the data. Consequently, this technology remains limited to displaying separate analysis results overlaid on the same area, and has the disadvantage of not being able to compensate for the shortcomings of drones (inability to cover a wide area) and satellites (inability to recognize that multiple plants are distributed in a single pixel).

[0025] As described above, most technologies either utilize images taken by drones or UAVs, or images taken by satellites, and both have inherent drawbacks. Specifically, technologies that utilize images taken by drones or UAVs are not suitable for wide-area remote sensing, while technologies that utilize images taken by satellites have the problem of difficulty in properly classifying plants when multiple types of plants are mixed in a field.

[0026] Therefore, the present invention aims to provide a vegetation discrimination system that combines the advantages of both drones or UAVs and satellite data. More specifically, the aim is to provide a vegetation discrimination system that combines the advantages of high-resolution images taken by drones or UAVs close to the Earth's surface (however, it is difficult to generate wide-area training data) and relatively low-resolution images taken by satellites far from the Earth's surface (however, it is easy to acquire data over a wide area).

[0027] Furthermore, the present invention aims to generate a predictive model for vegetation over a wide area by applying the estimation results from a machine learning model, which is generated by accurately training a machine learning model using high-resolution images taken by a drone or UAV in a relatively narrow area, to images taken by satellite, thereby enabling costly predictions about vegetation over a wide area.

[0028] Furthermore, this invention aims to overcome the drawback that plant distribution within a single pixel in satellite images cannot be associated with multiple plant species, and to estimate the vegetation cover of multiple plant classifications within a single pixel.

[0029] To achieve the above objective, the first invention is an information processing system for generating a learning dataset, which uses the estimation result, which is the output result of a trained model generated by machine learning using high-resolution data captured by a drone or UAV on a predetermined target area, as training data, and low-resolution data captured by satellite on a predetermined target area as input data, comprising: a high-resolution vegetation classification learning dataset generation unit that generates a high-resolution vegetation classification learning dataset by associating high-resolution data captured by a drone or UAV on a target area with training data indicating the vegetation classification of the target area; a first learning processing unit that performs machine learning processing to determine the vegetation classification from high-resolution data captured by a drone or UAV on a target area using the generated high-resolution vegetation classification learning dataset; and a high-resolution vegetation classification image generation unit that estimates the vegetation classification of the target area and outputs a high-resolution vegetation classification image using a first trained model generated by the first learning processing unit. The system is characterized by comprising: a vegetation ratio map generation unit that generates a vegetation ratio map, which is a map showing the vegetation ratio, which is the proportion of the ground surface covered by a predetermined plant or the proportion of bare ground, based on the high-resolution vegetation classification image generated, and which is a vegetation ratio map obtained by converting the low-resolution data captured by the satellite to a predetermined low resolution according to the resolution and dividing it into a grid.

[0030] The second invention is an information processing system for generating training data, characterized in that, in the information processing system described in the first invention, the system comprises a low-resolution vegetation ratio training data generation unit that generates a low-resolution vegetation ratio training dataset, which associates low-resolution data of the target area photographed by the satellite with a vegetation ratio map of the target area generated by the vegetation ratio map generation unit as training data.

[0031] The third invention is an information processing system for learning, characterized in that, in the information processing system described in the second invention, it further comprises a second learning processing unit that performs machine learning processing to determine the vegetation ratio from low-resolution data captured by a satellite, using a low-resolution vegetation ratio learning dataset obtained by associating the vegetation ratio map of the target area generated by the vegetation ratio map generation unit as training data with the low-resolution data of the target area captured by the satellite.

[0032] The fourth invention is an information processing system for vegetation discrimination, characterized in that, in the information processing system described in the third invention, the second learning processing unit performs machine learning processing using a low-resolution vegetation ratio learning dataset obtained by associating low-resolution data of the target area photographed by the satellite with a vegetation ratio map of the target area generated by the vegetation ratio map generation unit as training data, thereby generating a second trained model, and the low-resolution data photographed by the satellite is input to the second trained model, and a classifier is provided that determines the vegetation ratio of the target area based on the output from the second trained model.

[0033] The fifth invention is an information processing system described in the first to fourth inventions, characterized in that, in the vegetation ratio map which is training data generated by the vegetation ratio map generation unit, the label value indicating the correct answer is shown as a vegetation ratio that includes one or more of the following: the ratio of plants covering the ground surface, the grass ratio (or grass ratio, legume ratio), the grass ratio (or grass ratio, grass weed ratio), the legume ratio (or legume ratio, legume weed ratio), the weed ratio (or grass weed ratio, legume weed ratio, other weed ratio), and the bare ground ratio.

[0034] The sixth invention is a trained model for a computer to determine the vegetation of a target area based on low-resolution data captured by a satellite, wherein the second training unit performs machine learning processing using a training dataset in which the vegetation ratio map of the target area generated by the vegetation ratio map generation unit is associated with the low-resolution data of the target area captured by the satellite as training data, and the second training unit performs machine learning processing.

[0035] The seventh invention is a second trained model of the sixth invention, characterized in that the output of the second trained model is expressed as a vegetation ratio that includes one or more of the following: the percentage of the ground surface covered by plants, the percentage of pasture grass (or percentage of grass pasture grass, percentage of leguminous grass), the percentage of grasses (or percentage of grass pasture grass, percentage of grass weeds), the percentage of legumes (or percentage of leguminous grass, percentage of leguminous weeds), the percentage of weeds (or percentage of grass weeds, percentage of leguminous weeds, percentage of other weeds), and the percentage of bare ground.

[0036] The eighth invention is an information processing method for generating a learning dataset, in which the estimation result, which is the output result of a trained model generated by machine learning using high-resolution data captured by a drone or UAV on a predetermined target area, is used as training data, and low-resolution data captured by satellite on a predetermined target area is used as input data, the method comprising: a high-resolution vegetation classification learning dataset generation step of generating a high-resolution vegetation classification learning dataset by associating high-resolution data captured by a drone or UAV on a target area with training data indicating the vegetation classification of the target area; a first learning processing step of performing machine learning processing to determine the vegetation classification from high-resolution data captured by a drone or UAV on a target area using the generated high-resolution vegetation classification learning dataset; and a high-resolution vegetation classification image generation step of estimating the vegetation classification of the target area and outputting a high-resolution vegetation classification image using a first trained model generated by the first learning processing unit. The information processing method is characterized by comprising: a vegetation ratio map generation step, which generates a vegetation ratio map that shows the proportion of the ground surface covered by a predetermined plant or the proportion of bare ground, based on the high-resolution vegetation classification image generated, and which is obtained by converting the low-resolution data captured by the satellite to a predetermined low resolution and dividing it into a grid.

[0037] The ninth invention is an information processing method for generating training data, characterized in that, in the information processing method described in the eighth invention, the method comprises a low-resolution vegetation ratio training dataset generation step, which generates a low-resolution vegetation ratio training dataset, by associating the vegetation ratio map of the target area generated by the vegetation ratio map generation step with low-resolution data of the target area photographed by the satellite, as training data.

[0038] The tenth invention is an information processing method for learning, characterized in that, in the information processing method described in the ninth invention, it further comprises a second learning processing step in which a machine learning process is performed to determine the vegetation ratio from the low-resolution data captured by the satellite, using a low-resolution vegetation ratio learning dataset obtained by associating the vegetation ratio map of the target area generated in the vegetation ratio map generation step with the low-resolution data captured by the satellite as training data.

[0039] The eleventh invention is an information processing method described in the tenth invention, characterized by comprising: a step of generating a trained model by performing machine learning processing in the second learning processing step using a low-resolution vegetation ratio learning dataset obtained by associating low-resolution data of the target area photographed by the satellite with a vegetation ratio map of the target area generated by the vegetation ratio map generation unit as training data; and a classification step of inputting the low-resolution data photographed by the satellite into the trained model and determining the vegetation of the target area based on the output from the trained model.

[0040] The twelfth invention is an information processing method of the eighth to eleventh inventions, characterized in that, in the vegetation ratio map which is training data generated by the vegetation ratio map generation unit, the label value indicating the correct answer is shown as a vegetation ratio that includes one or more of the following: the ratio of plants covering the ground surface, the grass ratio (or grass ratio, legume ratio), the grass ratio (or grass ratio, grass weed ratio), the legume ratio (or legume ratio, legume weed ratio), the weed ratio (or grass weed ratio, legume weed ratio, other weed ratio), and the bare ground ratio.

[0041] The 13th invention is a method for generating a learned model for causing a computer to discriminate vegetation in a target area based on low-resolution data photographed by a satellite. In the information processing method described in the 10th invention, machine learning processing is performed in the second learning processing step using a low-resolution vegetation ratio learning data set in which a vegetation ratio map of the target area generated in the vegetation ratio map generation step is associated with the low-resolution data obtained by photographing the target area by the satellite, thereby generating a second learned model.

[0042] The 14th invention is a method for generating a second learned model of the 13th invention. In the vegetation ratio map, which is the teacher data generated by the vegetation ratio map generation unit, the label value indicating the correct answer is represented by a vegetation ratio including any one or more of the ratio of plants covering the ground surface, the grassland ratio (or the grass ratio of the Poaceae family, the grass ratio of the Fabaceae family), the Poaceae ratio (or the grass ratio of the Poaceae family, the weed ratio of the Poaceae family), the Fabaceae ratio (or the grass ratio of the Fabaceae family, the weed ratio of the Fabaceae family), the weed ratio (or the weed ratio of the Poaceae family, the weed ratio of the Fabaceae family, other weed ratios), and the bare land ratio. This is a method for generating a second learned model.

[0043] The 15th invention is a program for causing a computer to execute the information processing method according to any one of the 8th to 11th inventions.

[0044] The 16th invention is a program for causing a computer to execute the information processing method described in the 12th invention.

[0045] The 17th invention is an information processing system for generating a learning dataset, which uses an estimation result, which is the output result of a trained model generated by performing machine learning using high-resolution data of a predetermined target area, as training data, and low-resolution data of a predetermined target area as input data, comprising: a high-resolution attribute classification learning data generation unit that generates a learning dataset in which training data indicating the attribute classification of a predetermined target area is associated with high-resolution data of a predetermined target area; a first learning processing unit that performs machine learning processing to determine the attribute classification of a target area from high-resolution data of a predetermined target area using the generated high-resolution attribute classification learning dataset in which training data indicating the attribute classification of a predetermined target area is associated with high-resolution data of a predetermined target area; a high-resolution attribute classification image generation unit that estimates the attribute classification of the target area and outputs a high-resolution attribute classification image using a first trained model generated by the first learning processing unit; and an attribute ratio map generation unit that generates an attribute ratio map, which is a map showing the ratio of a predetermined attribute of the target area, converted to a predetermined low resolution according to the resolution of the low-resolution data of the predetermined target area and divided into a grid-like section, based on the generated high-resolution attribute classification image. The information processing system is characterized by comprising: a low-resolution attribute ratio learning dataset generation unit that generates a low-resolution attribute ratio learning dataset by associating the attribute ratio map of the target area, generated by the attribute ratio map generation unit, with the low-resolution data of the target area photographed in the aforementioned target area as training data; and a low-resolution attribute ratio learning dataset generation unit.

[0046] The 18th invention is an information processing system according to the 17th invention, further comprising: a second learning processing unit that performs machine learning processing to determine a ratio related to a predetermined attribute of a target area from low-resolution data obtained by photographing the target area, using a low-resolution attribute ratio learning data set in which an attribute ratio map of the target area generated by the attribute ratio map generation unit is associated with the low-resolution data of the target area as teacher data; a second learned model is generated by the second learning processing unit performing machine learning processing using a low-resolution attribute ratio learning data set in which an attribute ratio map of the target area generated by the attribute ratio map generation unit is associated with the low-resolution data of the target area as teacher data; and a classifier that inputs the low-resolution data obtained by photographing the target area into the second learned model and determines a ratio related to a predetermined attribute of the target area based on the output from the second learned model. The information processing system is characterized by comprising the above.

[0047] According to the present invention, in a relatively narrow area, high-precision annotation regarding vegetation is performed using high-definition image data captured by a drone or UAV, and then trained accurately. The estimation result from the learned model thus generated is used as the next-stage teacher data. Then, the next-stage learning process is performed using a learning data set combining the teacher data and the image data captured by the satellite, to generate a prediction model regarding vegetation over a wide area, making it possible to predict vegetation over a wide area without incurring costs.

[0048] Also, according to the present invention, it is possible to overcome the drawback that the plant distribution within one pixel in an image captured by a satellite cannot be associated with a plurality of plants, and to estimate the vegetation rate of a plurality of plant classifications within one pixel.

[0049] This figure shows an example of prior art. This figure shows an example of the overall system configuration of the present invention. This figure shows an example of the difference between the prior art and the present invention, where in the prior art, one plant classification result was assigned to each pixel of an image taken by a satellite, making it impossible to grasp that multiple plants are distributed in one pixel, whereas in the present invention, it is possible to show the vegetation cover rate of multiple plant classifications for one pixel. This figure shows the processing on the satellite side, and shows an example of the processing flow of satellite images. This figure shows the processing on the drone side, and shows an example of the overall overview flow from generating a satellite-resolution vegetation ratio map (vegetation cover map) from drone images to generating a low-resolution vegetation ratio learning dataset. This figure shows the processing on the drone side, and shows an example of the processing flow for performing segmentation processing of the target area. This figure shows the processing of the learning phase on the drone side, and shows an example of the processing flow when RF (Random Forest) is used in the machine learning processing (first learning processing) of vegetation classification. This figure shows the processing of the learning phase on the drone side, and shows an example of the processing flow when deep learning is used in the machine learning processing (first learning processing) of vegetation classification. This diagram shows the processing of the inference phase on the drone side, and is an example of the processing flow of estimation processing using unsupervised data with the SLIC-RF model or other machine learning models. This diagram shows an example of the processing flow of deep learning (second learning process) after data merging. This diagram shows an example of the processing flow of the inference phase in operation using the second trained model after deep learning (second learning process), and is an example of the overall processing flow of the system that performs vegetation discrimination from satellite images. This diagram shows an example of the image of the learning process when a one-dimensional convolutional neural network (1D-CNN) is used as the learning model in the second learning process. This diagram shows an example of the configuration of a one-dimensional convolutional neural network (1D-CNN). This diagram shows an example of the overview of the process of creating a vegetation ratio map (vegetation cover map) from vegetation classification images. This diagram shows an example of the details of the process of creating a vegetation ratio map (vegetation cover map) from vegetation classification images.Figure 15 shows an example of the process for generating a vegetation ratio map (vegetation cover map) showing the percentage of pasture grass from vegetation classification images classified as pasture grass. This figure shows an example of the details of satellite data composed of multispectral images (MS images), specifically an example for Landsat 8 / 9 (7 bands, resolution 30m). This figure shows an overview of satellite data composed of multispectral images (MS images), specifically an example for major optical satellites. This figure shows an example of the table structure of satellite data. This figure shows an example of performance evaluation (confusion matrix) of training data, which is the estimation result by the first trained model generated from images taken by a drone or UAV. This figure shows the performance evaluation results of the second trained model when using images from the Sentinel-2 satellite. Figure 20(A) shows an example of the mean and standard deviation of R2 and RMSE after repeating k-fold cross-validation (k=5) 10 times with the second trained model. Figure 20(B) shows the relationship between the measured and predicted values ​​of the Test data that was not used to train the model. This figure shows an example of the performance evaluation results of the vegetation ratio map (vegetation cover map) of the second pre-trained model using drone (measured values) and satellite data (predicted values) when using images from the Sentinel-2 satellite. This figure shows the performance evaluation results of the estimation results by the second pre-trained model when using images from the Landsat satellite. Figure 22(A) shows an example of the mean and standard deviation of R2 and RMSE after repeating k-fold cross-validation (k=5) 10 times with the second pre-trained model. Figure 22(B) shows the relationship between measured and predicted values ​​of the Test data that was not used to train the model. This figure shows an example of the performance evaluation results of the vegetation ratio map (vegetation cover map) of the second pre-trained model using drone (measured values) and satellite data (predicted values) when using images from the Landsat satellite. This figure shows the performance evaluation results of the estimation results by the second pre-trained model when using images from the PlanetScope satellite. Figure 24(A) shows an example of the mean and standard deviation of R2 and RMSE after repeating k-fold cross-validation (k=5) 10 times with the second trained model.Figure 24(B) shows the relationship between measured and predicted values ​​of the Test data that was not used for model training. This figure shows an example of the performance evaluation results of the vegetation ratio map (vegetation cover map) of the second trained model's drone (measured values) and satellite data (predicted values) when using PlanetScope satellite images. This figure shows an example of the system configuration. This is a functional block diagram of each processing unit.

[0050] ◆Explanation of Terms <Terms common to all inventions from the 1st to the 18th> ◇Target area refers to the area that is the subject of evaluation or analysis. ◇Field plot refers to a plot of land such as a farm or pasture, and is a plot of land from which forests, roads, rivers, etc. have been excluded from a designated farm or pasture. ◇Drone or UAV (Unmanned Aerial Vehicle) is a common name for an unmanned aerial vehicle that can be flown by remote control or autopilot, among other things such as airplanes, rotary-wing aircraft, gliders, and airships that are structurally incapable of carrying a person. ◇UAV image refers to an image taken by sensors or cameras mounted on an unmanned aerial vehicle such as a drone or UAV, but also includes images taken by a manned helicopter or small aircraft.

[0051] ◇UAV resolution refers to the high resolution obtained by taking images at a predetermined speed and altitude using high-resolution sensors or cameras such as 2K, 4K, or 8K mounted on a drone or UAV. It refers to a high resolution where one pixel corresponds to a small area on the ground such as 1cm x 1cm, 2cm x 2cm, 3cm x 3cm, etc. ◇Satellite resolution refers to the resolution obtained when taking images of the ground using sensors or cameras mounted on a satellite. It refers to a lower resolution where one pixel corresponds to a somewhat wider area on the ground such as 50cm x 50cm, 1m x 1m, 3m x 3m, 10m x 10m, 30m x 30, etc.

[0052] ◇A segment refers to a region that is divided into a single unit from which areas of vegetation distribution that are judged to be the same plant are grouped together. ◇Segmentation refers to the process of dividing areas of vegetation distribution that are judged to be the same plant into a single unit. When using the SLIC algorithm, the zero-parameter version of SLIC can be used, and the segment size can be set to, for example, 1 cm to 0.5 m. In this case, segments of an average size of 1 cm x 1 cm or 0.5 m x 0.5 m are formed, and this becomes one unit for classification.

[0053] ◇Ground Sampling Distance (GSD) is an index that indicates the number of pixels per unit area (usually in meters) on the ground, and is determined by the flight altitude and the camera's focal length. The higher the flight altitude, the wider the area of ​​the ground that one pixel can cover, resulting in a larger GSD, but lower accuracy.

[0054] ◇ A Geographic Information System (GIS) is a general term for system technologies that process and manage data containing various location-related information, and perform tasks such as map creation and advanced analysis. It is a system that overlays multiple data sets on a map and displays them in a visually easy-to-read format. ◇ A satellite refers to an artificial satellite, an artificial celestial body that exists in Earth's orbit and has a specific purpose.

[0055] ◇Satellite imagery refers to images taken by sensors and cameras mounted on artificial satellites, with a resolution of approximately tens of centimeters to tens of meters per pixel on the Earth's surface. Compared to high-resolution UAV imagery, it is considered low-resolution. ◇Resampling refers to calculating new pixel values ​​from the pixel values ​​of the original image.

[0056] ◇A spectrum is a record of the intensity distribution of electromagnetic waves (light) at different wavelengths, obtained by passing them through a spectrometer (such as a prism or diffraction grating). Depending on the mechanism by which the spectrum is produced, it is called a reflection spectrum, absorption spectrum, emission spectrum, or scattering spectrum. ◇A reflection spectrum is the spectrum of light reflected by a substance when light is irradiated onto it. Since the wavelength of reflected light (absorbed light) differs depending on the substance being measured, it is used to analyze the type and composition of substances.

[0057] ◇Spectral reflectance characteristics refer to the fact that the intensity of electromagnetic wave reflection varies depending on the wavelength band of the material, depending on the type of material on the Earth's surface. For example, "water" tends to reflect less as the wavelength increases, "soil" tends to reflect more as the wavelength increases, and "plants" are known to absorb visible electromagnetic waves (weakening their reflection) and strongly reflect near-infrared electromagnetic waves due to the action of photosynthetic pigments. This makes it possible to accurately determine the distribution of vegetation and vegetation cover on the Earth's surface.

[0058] ◇Multispectral refers to a spectrum measured using sensors for multiple wavelength bands. More specifically, it means multiple spectra observed across multiple different wavelength bands, including invisible light such as ultraviolet, infrared, and far-infrared rays that are invisible to the human eye. ◇Wavelength refers to the length of one wave cycle, i.e., the distance between peaks. Frequency is the number of waves per second and is expressed in Hertz (Hz). ◇Wavelength band refers to a predetermined range centered on a certain wavelength. In this specification and drawings, wavelength bands may be abbreviated as "BAND" or "B".

[0059] ◇A multispectral image (MS image) is an image that records electromagnetic waves in multiple wavelength bands. Multispectral images record not only electromagnetic waves in the visible light wavelength band that are visible to the human eye, but also electromagnetic waves in the invisible light wavelength band that are invisible to the human eye, such as ultraviolet rays, infrared rays, and far infrared rays. Sensors mounted on drones and satellites can detect electromagnetic waves and light in a certain range of wavelength bands. For example, in the case of Landsat 8 / 9 (7 bands, resolution 30 m), sensors for multiple wavelength bands are mounted, such as band 1 (Coastal / Aerosol, wavelength band 433-453 nm), band 2 (Blue, wavelength band 450-515 nm), etc., and the captured image constitutes a multispectral image (MS image). In this invention, since it deals with the types of plants and vegetation cover on the ground surface, a multispectral image mainly refers to data that records the reflection spectrum of an object across multiple wavelength bands.

[0060] ◇A hyperspectral image is an image that records electromagnetic waves across multiple wavelength bands, and in which case the number of corresponding wavelength bands is very large. Generally, in the case of multispectral imaging, the number of wavelength bands is about 10, while in hyperspectral images it is said to be about 100 bands. However, in this invention, a hyperspectral image is included in the term multispectral image (MS image), because both are images that record electromagnetic waves across multiple wavelength bands.

[0061] ◇ The HSV color space is a method of representing color using three elements: Hue (H), Saturation (S), and Value (V). The HSV color space can be converted from the RGB color space. In the HSV color space, the H value represents the type of color (red, green, blue, etc.). S and V represent vividness and brightness, respectively, and do not affect the type of color. Therefore, when detecting green using the HSV color space, you only need to focus on the H value. Generally, H takes values ​​within the range of 0 to 360, and values ​​in the range of 50 to 150 are considered green.

[0062] ◇The Lab color space is a type of complementary color space that has a dimension L representing lightness and complementary color dimensions a and b, and is based on a non-linear compression of the coordinates of the CIE-XYZ color space. In the Lab color space, L represents the brightness of the color, a represents the hue of red and green, and b represents the hue of yellow and blue. Therefore, like the HSV color space, the Lab color space is considered effective for detecting green because there is only one parameter that governs green. When a takes a large negative value, the green becomes stronger, when it takes a large positive value, the red becomes stronger, and when it is 0, it becomes achromatic. Therefore, by specifying a < 0, it becomes possible to detect the green region.

[0063] ◇Hue conversion (color space conversion) is a conversion technique used to represent the color information of an image using a different representation method. A common example of hue conversion (color space conversion) is the conversion from RGB to HSV (Hue, Saturation, Value). RGB (Red, Green, Blue) is the three primary colors of light, and the color information of an image is represented by the intensity of red, green, and blue. However, in the RGB color space, it can be difficult to directly grasp the characteristics of color such as hue and saturation, making it unsuitable for distinguishing plant species, for example. In contrast, in the HSV color space, the type of color (red, green, blue, etc.) is determined by a single value, H. Similarly, in the Lab color space, green is determined by a single value, a. Therefore, detecting green is more effective when performed in the HSV or Lab color space.

[0064] For this reason, hue conversion (color space conversion) is widely used in image processing AI for color-related tasks and applications, such as image segmentation, color detection, color correction, and color-based feature extraction. The optimal color space to use varies depending on the object to be classified, so in this invention, the HSV color space was used as an example, but it is not limited to this. In the preliminary analysis of this invention, a comparison was made between RGB images and HSV images in the classification process of plants containing a large amount of green. When classification was performed using the same training data with RGB images, the overall accuracy rate was 0.864, while with HSV images the accuracy rate was 0.919, which is about 0.055 (5.5%) higher, indicating that the conversion effect of HSV is significant. The reason for this is thought to be that RGB images tend to have high correlation between bands, and the amount of information when classifying segments in this case is small, whereas by converting to an HSV image, the correlation between bands decreases, and the amount of information in classification increases.

[0065] ◇Orthorectification is a process that converts aerial photographs (UAV images) taken with central projection into orthorectified images, so that an orthorectified image looks like a photograph viewed from directly above, regardless of the viewing point. ◇Vegetation refers to a group of plants growing together in a particular area.

[0066] ◇The vegetation classification of the target area refers to information indicating the classification of the vegetation in the target area. Vegetation classifications include pasture grasses, weeds, bare ground, as well as classifications such as deciduous broad-leaved forests, evergreen coniferous forests, shrub communities, herbaceous plant communities, cultivated land, and non-vegetated areas, or classifications such as cedar, cypress, bamboo forests, pine, broad-leaved trees, water areas, shrubs and others, or classifications related to plants such as vegetation cover (VC), trees (tall trees and shrubs), community height and vegetation height, biomass (vegetation cover × vegetation height), plant growth state, long-grass type, short-grass type, etc. In the case of pastures, it also includes detailed classifications such as grasses (GR), legumes (LG), grass weeds (GW), other weeds (MW), and bare ground (BG). Note that vegetation classification is intended to be performed for each pixel.

[0067] ◇Annotation means "commentary" or "annotation." It is synonymous with labeling. In the IT field, it refers to the process of attaching information called tags or metadata to each individual piece of data, whether it be text, audio, images, or videos. In the field of AI and machine learning, it refers to the process of assigning training data or labels to the data to be analyzed. It is a process performed in supervised learning, which is one of the classifications of machine learning.

[0068] ◇Training data indicating the vegetation classification of the target area refers to data generated by annotating high-resolution data (UAV resolution) taken of the target area by a drone or UAV, indicating which vegetation classification each pixel belongs to, for example, in the case of pastureland, grasses (GR), legumes (LG), grass weeds (GW), other weeds (MW), or bare ground (BG). The training data is added after confirming plant classification and vegetation through field surveys, by visually identifying the data on the image (for example, an image after segmentation processing).

[0069] ◇The high-resolution vegetation classification learning dataset refers to a dataset for learning processing that associates high-resolution data (UAV resolution) taken of a target area by a drone or UAV with the aforementioned training data on vegetation classification. ◇The high-resolution vegetation classification learning dataset generation unit is a control unit that generates the aforementioned high-resolution vegetation classification learning dataset.

[0070] ◇SLIC (Simple Linear Iterative Clustering) is one of the algorithms involved in image recognition, and is a type of image segmentation method. SLIC is an algorithm that takes an image as input and outputs the division into superpixels, which are units of pixels that form a certain grouping, and partially applies the K-means algorithm. ◇RF (Random Forest) is an algorithm that combines two methods, "decision trees" and "ensemble learning (bagging)," and is used for applications such as "classification" in machine learning.

[0071] ◇SLIC-RF refers to an object-oriented image analysis processing technology that combines SLIC segmentation and Random Forest (RF). Because it can achieve high classification accuracy with relatively few parameters and computation time, it is suitable for discrimination processing such as distinguishing grass species and weeds in high-resolution images (which generally require a long processing time). Object-oriented image analysis technology is used as one of the image analysis methods for high-resolution images such as those obtained from drone aerial photography (UAV images). In UAV images, high-resolution images are obtained where one pixel (one image element) is about several centimeters, and plant bodies, droppings, etc. can be clearly distinguished. Object-oriented image analysis groups pixels with similar brightness based on specific rules and creates segments consisting of multiple pixels. This method extracts objects on the image by performing classification using these segments as units.

[0072] Object-oriented image analysis primarily consists of three processes: segmentation (creating segments), feature extraction for segments, and segment classification based on these features. Multiple analysis methods are available for each process, and these are combined to perform image processing, ultimately yielding a vegetation classification image. If necessary, the segmentation method (SLIC, deep learning), the metrics to be used as features, and the classification model for segment classification (RF, deep learning) are selected. While the embodiment of this invention shows an example using SLIC for segmentation and RF or deep learning for classification, it is not limited to this.

[0073] ◇The target region segment dataset (with training data) refers to a training dataset in which training data, which labels the vegetation classification of each segment, is added to segment data after the segmentation processing of UAV images of the target region has been completed. ◇The first training processing unit is a control unit that, after the segmentation processing of the target region is completed, performs machine learning (first training processing) based on the target region segment dataset (with training data) to generate a first trained model that performs classification processing for each segment. When using deep learning, both segmentation processing and classification processing of the target region may be performed to generate the first trained model.

[0074] ◇The first learning process refers to the process executed by the first learning processing unit. ◇The first trained model refers to the trained machine learning model generated by machine learning through the first learning process. Vegetation classification processing is performed on the segmented images (SLIC, etc.) of the HSV images generated by preprocessing UAV images, and the estimated result of the vegetation classification image (UAV resolution) for the target area is output.

[0075] ◇A vegetation classification image is an image (high-resolution image at UAV resolution) showing the estimated results of vegetation classification output by the first trained model. In the case of pastures, classification is performed for grasses (GR), legumes (LG), grass weeds (GW), other weeds (MW), bare ground (BG), etc., and classification can be performed pixel by pixel. ◇The high-resolution vegetation classification image generation unit is a control unit that generates high-resolution vegetation classification images (UAV resolution) related to vegetation classification using the first trained model based on high-resolution UAV images without training data.

[0076] ◇Vegetation ratio (vegetation cover rate) refers to the proportion of the ground surface covered by a given plant or the proportion of bare ground (including cases where both are present). It may also refer to the proportion of the ground surface covered by any plant, rather than a specific plant. It is synonymous with vegetation-coverage and can also be defined as the unit area of ​​plant communities projected onto the ground surface. Generally, when viewed over a small area, it is often sufficient to consider the area as covered by a single plant or bare ground, but as the area widens, it is desirable to evaluate it using the proportion of the area covered by a given plant or the proportion of the area that is bare ground. More specifically, in the case of UAV resolution (several cm x several cm), one vegetation classification corresponds to one pixel, whereas in the case of satellite resolution (tens of cm to tens of m), multiple vegetation classifications correspond to one pixel, so it is possible to represent the vegetation ratio (vegetation cover rate) of multiple plant classifications for one pixel (tens of cm to tens of m) of a satellite image.

[0077] ◇A vegetation ratio map (vegetation cover map) is an image that shows the vegetation ratio (vegetation cover rate). The vegetation ratio map is an image that is converted from a vegetation classification image (UAV resolution) output by the first trained model into a format that represents the composition ratio of each vegetation classification when multiple pixels are aggregated and replaced with one pixel at satellite resolution based on grid information at satellite resolution. In a vegetation classification image, one vegetation classification can be represented per pixel, whereas in a vegetation ratio map, the vegetation ratio (vegetation cover rate) of multiple plant classifications can be represented per pixel. ◇The vegetation ratio map generation unit is a control unit that performs the process of generating a vegetation ratio map (vegetation cover map) corresponding to satellite resolution based on a vegetation classification image (UAV resolution) output by the first trained model.

[0078] ◇The low-resolution vegetation ratio training dataset refers to a dataset that associates low-resolution data (satellite images) of a target area taken by satellite with a vegetation ratio map of the same area generated by the vegetation ratio map generation unit, using these as training data. This is the training dataset used in the second training process.

[0079] ◇The second learning processing unit is a control unit that performs machine learning processing using high-resolution UAV images taken by a drone or UAV as input data, and relatively low-resolution satellite-resolution MS images, and uses the vegetation ratio map (vegetation cover map), which is the prediction result output by the first trained model, as training data. Various machine learning algorithms can be used, including deep learning. ◇The second learning process refers to the learning process of the second learning processing unit. ◇The second trained model refers to the trained machine learning model generated by machine learning through the second learning process. Based on satellite images (low-resolution satellite-resolution data) of the target area to be analyzed, it performs vegetation discrimination processing and outputs the estimation result of the vegetation ratio map (vegetation cover map).

[0080] ◇Convolutional Neural Networks (CNNs) are deep neural networks primarily used in image recognition. By leveraging features such as local receptive fields and weight sharing, they enable high-precision recognition of spatial features with fewer parameters compared to fully connected neural networks. This is an example of a machine learning model used when deep learning is employed in the second learning process.

[0081] ◇A one-dimensional convolutional neural network (1D-CNN) is a convolutional neural network that uses a one-dimensional convolutional filter, whereas convolutional neural networks generally use a two-dimensional convolutional filter. While two-dimensional convolutional filters extract features from images, one-dimensional convolutional filters extract feature patterns from one-dimensional data such as time-series data. Multispectral images (MS images) captured by multispectral cameras mounted on satellites are one-dimensional graph data captured by sensors in multiple wavelength bands, making them well-suited for processing by one-dimensional convolutional neural networks (1D-CNNs).

[0082] <Terminology for Inventions 17-18> ◇Attribute ratio refers to an indicator of how much of the attributes of a target area can be quantified within one pixel, including the vegetation ratio (vegetation cover rate) of multiple plant classifications, as well as the land use status of fields, farms, pastures, fallow land, etc. More specifically, in the original high-resolution data, one pixel corresponds to, for example, one plant classification or one land use status, whereas in the low-resolution data obtained by aggregating the attributes of multiple high-resolution pixels and converting them to low resolution, one pixel can represent the vegetation ratio of multiple plant classifications or the land use ratio of multiple types.

[0083] ◇The attribute classification of the target area is a concept that extends the vegetation classification of the target area, and refers to a classification derived based on various information regarding the attributes of the target area, including not only the vegetation classification of the target area but also the plant classification, as well as land use conditions such as forests, wasteland, rice paddies, fields and other agricultural land, pastures, fallow land, land under construction, vacant land, industrial land, general low-rise residential areas, densely populated low-rise residential areas, medium- and high-rise residential areas, commercial and business land, road land, parks and green spaces, other public and public-interest facility land, rivers and lakes, etc. (See Invention 17).

[0084] ◇Training data indicating attribute classification of the target area is an extended concept of training data indicating vegetation classification of the target area. It refers to data generated by annotating classifications derived from various information about the attributes of the target area, such as land use status (paddy fields, cultivated fields and other agricultural land, pastures, fallow land, etc.), in addition to training data indicating vegetation classification of the target area. ◇Segment dataset of the target area (with training data) refers to a dataset for learning processing in which training data, which labels the attribute classification of a segment, is attached to segment data after the segmentation processing of high-resolution images of the target area.

[0085] ◇The first learning processing unit is a control unit that, after completing the segmentation processing of the target region, performs machine learning (first learning processing) based on the segment dataset (with training data) of the target region to generate a first trained model that performs classification processing for each segment. When using deep learning, both segmentation processing and classification processing of the target region may be performed to generate the first trained model. ◇The first learning processing refers to the processing performed by the first learning processing unit. ◇The first trained model refers to the trained machine learning model generated by machine learning through the first learning processing. High-resolution images that have undergone preprocessing, etc., are segmented by segmentation processing (SLIC, etc.), and attribute classification processing is performed on the segmented images, and the estimation results of the attribute classification images (high-resolution images) of the target region are output.

[0086] ◇The high-resolution attribute classification learning dataset is an extended concept of the high-resolution vegetation classification learning dataset, and refers to a learning processing dataset that associates high-resolution data (UAV resolution) taken of a target area by a drone or UAV with the aforementioned training data on vegetation classification, as well as a learning processing dataset that associates training data generated by annotating classifications derived based on various information about the attributes of the target area, such as land use status of rice paddies, fields and other agricultural land, pastures, fallow land, etc. (See Invention 17). ◇The high-resolution attribute classification image generation unit is a control unit that generates the high-resolution attribute classification learning dataset.

[0087] ◇An attribute ratio map is an image that shows attribute ratios. A vegetation ratio map is an image that is converted from an attribute classification image (high resolution) output by the first trained model to a format that represents the composition ratio of each attribute classification when multiple pixels are aggregated and replaced with one low-resolution pixel based on low-resolution grid information. In an attribute classification image, one attribute classification can only be represented per pixel, whereas in an attribute ratio map, the attribute ratios of multiple attribute classifications can be represented per pixel.

[0088] ◇The attribute ratio map generation unit is a control unit that generates a low-resolution attribute ratio map based on the attribute classification image (high resolution) output by the first trained model. ◇The low-resolution attribute ratio training dataset is a dataset that associates low-resolution data of a target area with the attribute ratio map of the target area generated by the attribute ratio map generation unit as training data. This is the training dataset used in the second training process.

[0089] ◇The second learning processing unit is a control unit that takes relatively low-resolution images as input data to high-resolution images of the target area captured by a high-resolution sensor, and uses the attribute ratio map, which is the prediction result output by the first trained model, as training data to perform machine learning processing. Various machine learning algorithms can be used, including deep learning. ◇The second learning process refers to the learning process of the second learning processing unit. ◇The second trained model refers to the trained machine learning model generated by machine learning through the second learning process. Based on the image of the target area to be analyzed (low-resolution data), it performs attribute discrimination processing and outputs the estimated result of the attribute ratio map.

[0090] The following describes embodiments of the present invention. Note that the configurations, figures, and tables described are merely illustrative and can be applied to other shapes and configurations.

[0091] 1. Overall Overview of the Invention First, an overall overview of the vegetation discrimination processing system of the present invention will be explained using Figure 2. In Figure 2, to provide an overview of the whole, the learning phase, in which a trained model is generated by a learning process using "supervised data", and the inference phase, in which "unsupervised data" is input to the generated trained model and the discrimination result is output, are shown together. The processing flow separating the learning phase and the inference phase is described in the explanations in Figures 5 to 12.

[0092] 1-1. Satellite-side processing The target area is photographed by the satellite and satellite images are acquired (Step S1-1). After preprocessing such as resampling is performed on the satellite images (Step S1-2), a multispectral image is generated (Step S1-3). Details of the multispectral image (hereinafter sometimes abbreviated as "MS image") will be described later (see Figures 16 and 17). Next, the region of interest is extracted from the MS image (Step S1-4) to generate a region of interest image (hereinafter sometimes abbreviated as "ROI image") (Step S1-5). The region of interest refers to the part of the image that is particularly noteworthy for processing and analysis. Based on the field division information, grid information (satellite resolution) of the area to be evaluated is generated from the ROI image (Step S1-6).

[0093] The grid information (satellite resolution) generated by the satellite-side processing is transmitted to the drone-side processing (step S1-7), and together with the grid information (UAV resolution) generated by the drone-side processing, it is used to generate a vegetation ratio map (vegetation cover map) that shows the vegetation ratio (vegetation cover rate), which is the percentage of the ground surface covered by a given plant or the percentage of bare ground, from the classification image generated (estimated) in the first inference phase on the drone side.

[0094] 1-2. Drone-side processing (first learning process) The target area is photographed with a UAV or drone, and UAV images are acquired (step S2-1). Next, as a preprocessing step, the color space of the UAV images is converted, and the image processing is performed to convert it to a color space that makes it easy to distinguish vegetation for the desired plants (step S2-2). For example, HSV hue conversion is performed (step S2-3). Then, using the HSV images, segmentation processing is performed to divide the target area into segments of a predetermined size for each predetermined type of vegetation (first half of step S2-4). SLIC processing can be used as an example for segmentation processing, but is not limited to this.

[0095] The segment size can be selected to match the UAV resolution, for example, 1 cm x 1 cm, 3 cm x 3 cm, and various other sizes. Generally, it is known that prediction accuracy improves as the segment size decreases (see, for example, "Kensuke Kawamura, Taisuke Yasuda, Miya Kitagawa, Masato Yashiroda, Kyoko Kunishige (2023b) Relationship between drone flight altitude and ground resolution - Towards efficient aerial photography of large fields -, Japanese Society of Grassland Science 69(3): 138-144"), but when used as training data for satellite resolution, it is converted to vegetation cover of several tens of centimeters x several tens of centimeters, so it is acceptable to use a somewhat large size. For example, Table 2 in the aforementioned document shows that, when using a CCD (image sensor) with a predetermined number of pixels, the relationship between the flight altitude of a UAV, the camera's field of view, and the ground resolution (GSD) is as follows: at a flight altitude of 20m, the GSD is 0.67cm for a field of view of 90 degrees, 0.71cm for a field of view of 70 degrees, and 0.87cm for a field of view of 50 degrees. Furthermore, at a flight altitude of 30m, the GSD changes from 1.00cm to 1.07cm to 1.31cm in accordance with the change in field of view; at a flight altitude of 50m, the GSD changes from 1.67cm to 1.78cm to 2.18cm in accordance with the change in field of view; and at a flight altitude of 80m, the GSD changes from 2.67cm to 2.84cm to 3.49cm in accordance with the change in field of view. While prediction accuracy tends to improve with smaller GSD, the flight altitude and field of view of the CCD (image sensor) and UAV can be appropriately set so that the GSD is approximately 3 cm or less.

[0096] Next, the segment features are calculated, and the segments are classified based on these features (second half of step S2-4). The indicators used as features are those related to the color of the segment. Training data is obtained from field surveys, and annotations are applied to each region divided by the segmentation process to provide training data related to plant classification. The classification process is performed using a trained machine learning model generated by machine learning processes such as RF (Random Forest) or deep learning (first training phase).

[0097] Then, UAV images of the area to be evaluated, for which there is no training data, are acquired, and classification images and grid information (UAV resolution) related to the vegetation classification of the area to be evaluated are generated using the generated pre-trained machine learning model (first inference phase) (step S2-5). Furthermore, based on the classification images and grid information (UAV resolution) generated (estimated) in the first inference phase, and the grid information (satellite resolution) generated by the satellite-side processing, a vegetation ratio map (vegetation cover map) showing the vegetation ratio (vegetation cover rate), which is the proportion of the ground surface covered by a given plant or the proportion of bare ground, is generated (steps S2-6, S2-7).

[0098] 1-3. Second Learning Process and Operation of the Vegetation Discrimination System After the satellite-side processing and drone-side processing described above have been performed, a second learning dataset for the second learning process is generated (data merging process) (step S3-1). This dataset is created by linking the classification image related to vegetation classification, which is the output result (estimated result) of the first trained model generated in the first learning phase on the drone side, and the vegetation ratio map (vegetation cover map) showing the vegetation ratio (vegetation cover rate) generated from the classification image, with the MS image with grid information of the area to be analyzed, which was generated in the satellite-side processing, as training data.

[0099] Furthermore, a second machine learning process is performed using the second training dataset (steps S3-2, S3-3) to generate a second trained model. Then, satellite images (MS images) of the evaluation area, for which no training data has been assigned, are input to the generated second trained model, and estimation results regarding vegetation ratio (vegetation cover) are output (step S3-4).

[0100] 1-4. Summary As described above, the present invention is characterized in that, when performing machine learning processing (second learning process) to determine the vegetation ratio (vegetation cover rate) based on image data captured by a satellite, the output from a first trained model trained using images captured by a drone or UAV (estimated result for vegetation ratio (vegetation cover rate)) is used as training data.

[0101] Therefore, firstly, if training data for the minimum necessary area of ​​UAV images with sufficient accuracy can be prepared, a first pre-trained model with good accuracy can be generated. Secondly, if unsupervised UAV images without training data are prepared for a wide area and input into the first pre-trained model, it is possible to output estimation results for vegetation ratio (vegetation cover) with sufficient accuracy for a wide area. Thirdly, a second machine learning process can be performed using satellite images without training data (which can be obtained without effort) as input data, and the estimation results for vegetation ratio (vegetation cover), which are the output results of the first pre-trained model (which can be generated without effort), as training data. Fourthly, as a result, it is possible to obtain a second pre-trained model with sufficient accuracy.

[0102] Furthermore, in conventional technology, when generating a classifier that identifies vegetation using satellite imagery as input, the training process involved assigning one type of vegetation classification (such as water surface, grassland, bare ground, or forest) to each pixel of the satellite image as training data. As a result, the classifier's judgment results from the trained model were limited to outputting only one type of vegetation classification per pixel of the satellite image, which was inconvenient because it could not represent the diverse distribution of vegetation within the size of a single pixel (on the order of tens of centimeters to tens of meters).

[0103] In contrast, the classifier using the pre-trained model of the present invention utilizes training data on vegetation based on high-resolution UAV images for satellite imagery. This allows for training that uses the vegetation ratios (vegetation cover) of multiple plant classifications as training data for each pixel of satellite imagery, such as pasture percentage, weed percentage, and bare ground percentage. As a result, the classifier using the pre-trained model can output the vegetation ratios (vegetation cover) of multiple plant classifications for each pixel of satellite imagery, making it possible to represent the distribution of diverse vegetation within a single pixel size (on the order of tens of centimeters to tens of meters) (see Figure 3).

[0104] 2. Processing Flow of the Vegetation Discrimination Processing System Next, the processing flow of the vegetation discrimination processing system of the present invention will be described using Figures 4 to 11. Note that the program describing each processing step may be stored in the temporary storage unit (RAM, etc.) or hard disk of a computer, or it may be recorded on a tangible recording medium that is not temporary (CD-ROM, USB memory, etc.) (see Figure 26). 2-1. Satellite-side processing (generation of MS images and grid information) Figure 4 is a diagram showing an example of the processing flow when generating MS images with grid information that match the field divisions from satellite images (an example of the detailed contents of steps S1-1 to S1-7 of the processing flow in Figure 2). First, the satellite image of the target area is loaded from the satellite image DB of the storage unit (step S4-1).

[0105] Next, preprocessing such as resampling is performed on the satellite image (step S4-2) to generate an MS (multispectral) image (step S4-3). Details of the MS (multispectral) image will be described later, but it is an image recorded using a sensor mounted on the satellite that records electromagnetic waves in multiple wavelength bands, and it is an image of multiple wavelength bands for determining the type of object by utilizing the fact that the intensity of reflection of electromagnetic waves changes for each wavelength band depending on the type of object on the Earth's surface (see Figures 16 and 17).

[0106] Next, the field plot data of the area to be analyzed is loaded from the memory unit, and the image is cropped while aligning it with the field plot to be analyzed from the satellite image (MS image) (step S4-4), thereby generating a so-called ROI (region of interest) image (step S4-5). In addition, grid information tailored to the field plot is generated (step S4-6), and the MS image with grid information tailored to the field plot is saved to the memory unit (step S4-7). Note that the grid information generated by the satellite-side processing is at satellite resolution and is different from the grid information (UAV resolution) refined separately by the drone-side processing.

[0107] 2-2. Drone-side processing 2-2-1. Overview Figure 5 shows an example of the processing flow on the drone side, from performing machine learning using drone images to generate a trained model, to using the output of the generated trained model (predicted values ​​at drone resolution) to generate a satellite-resolution vegetation ratio map (vegetation cover map) (an example of detailed content of steps S2-1 to S2-7 of the processing flow in Figure 2).

[0108] More specifically, this shows an example of a processing flow that involves converting UAV images into HSV images in a color space suitable for vegetation classification, generating a classifier using a first pre-trained model that performs machine learning with the HSV images to perform vegetation classification, generating classification images related to vegetation classification using the first pre-trained model, generating a satellite-resolution vegetation ratio map (vegetation cover map) from the UAV-resolution classification images related to vegetation classification to use as training data, and generating a training dataset for the second training process described later.

[0109] In the flowchart of Figure 5, the first half, from steps S5-1 to S5-5, represents the learning phase; the middle section, from steps S5-6 to S5-8, represents the inference phase using the first trained model; and the latter half, from steps S5-9 to S5-12, represents the generation phase of the training dataset using the output results of the first trained model. Here, the cylinder on the right side of Figure 5 indicates the contents of the memory unit, and within it, two types of grid information are shown: "UAV resolution grid information" and "satellite resolution grid information." The distinction is that the grid with finer squares represents "UAV resolution grid information," and the grid with larger squares represents "satellite resolution grid information."

[0110] 2-2-2. Learning Phase Next, the learning phase process will be explained using the upper part of Figure 5. In the learning phase, first, UAV images of the target area are loaded from the UAV image DB in the memory unit (step S5-1). Next, preprocessing such as HSV hue conversion is performed on the UAV images (step S5-2) to generate HSV images (step S5-3). Details of HSV images will be described later, but by converting the color space, the images are converted into images with a color space that makes it easier to extract features related to vegetation, depending on the application, such as grassland analysis.

[0111] Next, machine learning processing (first training process) for vegetation classification is performed using HSV images and training data created by annotating vegetation classification (step S5-4). The machine learning processing can utilize the SLIC-RF algorithm or deep learning. Then, by performing machine learning processing using a predetermined dataset, a first trained model that estimates vegetation classification from HSV images at UAV resolution is generated and saved (step S5-5).

[0112] 2-2-3. Inference Phase Using the First Pre-trained Model and Training Dataset Generation Phase Using the Output Results of the First Pre-trained Model (1) Inference Phase Using the First Pre-trained Model Next, the processing of the inference phase using the first pre-trained model will be explained using the middle section of Figure 5. In the inference phase using the first pre-trained model, first, the first pre-trained model that estimates vegetation classification from HSV images is loaded to generate a classifier that estimates vegetation classification (step S5-6). Then, the UAV images (HSV images) of the evaluation target area that have been prepared in advance are loaded (step S5-7), input to the classifier that estimates vegetation classification by the first pre-trained model, and a vegetation classification image (estimated value) at UAV resolution is output (step S5-8).

[0113] Furthermore, the UAV images (HSV images) of the evaluation area used in the inference phase using the first trained model may include those with training data attached, but basically, they can be UAV images without training data attached, which is noteworthy because it eliminates the need for on-site surveys of vegetation (they can be obtained simply by flying a drone or UAV).

[0114] (2) Phase for generating a training dataset using the output results of the first trained model Next, the phase for generating a training dataset using the output results of the first trained model will be explained using the lower part of Figure 5. Based on the vegetation classification image at UAV resolution and the grid information at satellite resolution generated by the satellite processing, a vegetation ratio map (vegetation cover map) is generated by converting the target area to satellite image resolution and dividing it into a grid (step S5-9), and it is stored in the memory unit (step S5-10).

[0115] Here, the training data corresponding to the UAV-resolution HSV images and the vegetation classification images, which are estimation results from the first trained model, show one type of vegetation classification per pixel (see the left side of Figure 3). However, the vegetation ratio map (vegetation cover map) is transformed so that each pixel represents the vegetation ratio (vegetation cover rate) of multiple plant classifications (see the right side of Figure 3). The details of the process of generating a satellite-resolution vegetation ratio map (vegetation cover map) from UAV-resolution vegetation classification images will be described later, so they are omitted here (see Figure 14).

[0116] Then, a low-resolution vegetation ratio learning dataset is generated by associating low-resolution satellite-resolution data, obtained by photographing the target area by satellite, with a satellite-resolution vegetation ratio map as training data (step S5-11), and storing it in the memory unit (step S5-12). Note that the process in step S5-11 corresponds to the data merging process in step S3-1 in Figure 2. Furthermore, the processes from steps S5-1 to S5-9 described above are defined as the first invention, and the process in step S5-11 is defined as the second invention.

[0117] 2-2-4. Details of Segmentation and Classification Processing 2-2-4-1. Processing of the SLIC-RF Algorithm The SLIC-RF algorithm is described in detail in the literature listed in the remarks column of Technical A-1 in Figure 1, but for the sake of full disclosure of the invention, it will be explained below using Figures 6 and 7.

[0118] (1) Segmentation processing using SLIC as an example. Figure 6 shows a detailed example of the SLIC segmentation processing, which is a preliminary processing step to the SLIC-RF processing in step S2-4 of Figure 2 and step S5-4 of Figure 5. Segmentation processing is the process of dividing the target area into segments of a predetermined size according to predetermined vegetation. While the SLIC algorithm is used as an example for segmentation processing, other algorithms or various machine learning processes may also be used.

[0119] First, a UAV image is acquired from the UAV image database (step S6-1), and a color space conversion process is performed, such as converting the original RGB image to the HSV color space or the Lab color space (l, a, b) (step S6-2). The reason for changing the color space here is that in the case of RGB space, detecting green requires adjusting three parameters, R, G, and B, which is inefficient, whereas in the HSV color space, the type of color (red, green, blue, etc.) is determined by a single value, H, which is efficient and is expected to improve accuracy. Similarly, in the Lab color space, green is determined by a single value, a.

[0120] Furthermore, the optimal color space for identification varies depending on the target plant, and the HSV color space is closely linked to human color perception and is robust against brightness fluctuations, making it suitable for use with images taken in the field and suitable for segmenting green vegetation against a soil background. In the experiment of this invention, we will explain using the HSV color space as an example.

[0121] Next, as part of the first initialization process, the positions of the centroids of a predetermined number of k clusters are arranged at equal intervals (step S6-3). Then, as part of the second initialization process, the centroids are moved to the position with the minimum gradient by referring to the surrounding n x n pixels (for example, 3 x 3 pixels) neighborhood including each centroid (step S6-4). Next, local clustering (clustering with a narrowed range), which is local clustering, is performed on a portion of the image (step S6-5), and while moving the centroids (step S6-6), local clustering and centroid movement are repeated sequentially until a predetermined termination condition is met (step S6-7).

[0122] Here, the termination conditions include when the maximum number of iterations falls below 10, and when the Manhattan distance between the destination centroid and the current centroid falls below a threshold. Here, "iteration" means "repeatedly," and is the same as the batch size, which is the number of iterations in machine learning. Manhattan distance refers to the distance traveled while turning at right angles on a grid. Based on the above processing details, SLIC is positioned as a local clustering method.

[0123] Once clustering is complete, segment data is generated for the entire target area (step S6-8). The segment size can be appropriately selected depending on the target accuracy and purpose, but in this embodiment, a size of 1 cm x 1 cm was adopted, within the range of the UAV resolution. Then, annotation (labeling) is performed on the segment data of the target area, training data on plant classification obtained from field surveys is added (step S6-9), and the segment dataset of the target area (with training data) is saved (step S6-10).

[0124] Examples of training data for plant classification include grasses (GR), legumes (LG), grass weeds (GW), other weeds (MW), and bare ground (BG). Other possible labelings include vegetation cover (VC), canopy height, vegetation height, biomass (vegetation cover × vegetation height), trees (tall trees and shrubs), tall grass type, and short grass type.

[0125] (2) Example of classification processing using RF (Random Forest) Next, the processing of RF (Random Forest) will be explained using Figure 7. Figure 7 is a diagram showing the processing content of the learning phase of the machine learning processing (first learning processing) of vegetation classification, and shows the details of the processing flow of the latter half of step S2-4 in Figure 2 to step S5-4 in Figure 5. As an example of a learning algorithm, an example using the RF (Random Forest) algorithm will be shown.

[0126] First, the segment dataset (with training data) (UAV resolution) of the target region generated by the segmentation process is loaded (Step S7-1). Next, features are calculated for each segment (Step S7-2). Sixteen features were selected: four variables × four statistical values. More specifically, the median, standard deviation, maximum value, and minimum value of each band were calculated for each segment and used as statistical values ​​(four statistical values). Also, since the image to be analyzed has four bands, 16 features (four bands × four statistical values) are calculated per segment.

[0127] Next, classification processing is performed using the RF (Random Forest) algorithm (step S7-3). This process is positioned as training an RF model with features as explanatory variables. After completing the training process for all segments (step S7-4), the classifier generated by the RF (Random Forest) machine learning process (the first trained model) is stored in the memory unit (step S7-5).

[0128] 2-2-4-2. Examples of Segmentation and Classification Processing Using Deep Learning For classification processing, in addition to RF (Random Forest), other machine learning processes, such as deep learning, may be used. As an example, the case in which a trained model generated by deep learning is used for classification processing will be explained using Figure 8. Note that deep learning processing can also be used for the entire segmentation and classification processing (SLIC-RF processing). In that case, training processing should be performed by adding training data that includes segment information, as well as training data for vegetation classification, to the UAV images of the target area.

[0129] Figure 8 shows the processing details of the learning phase of the machine learning process (first learning process) for vegetation classification, and illustrates the processing flow in detail for the latter half of step S2-4 in Figure 2 to step S5-4 in Figure 5. As an example of a learning algorithm, an example using deep learning is shown. First, the learning model is set up on a machine learning processing platform (step S8-1). The settings include the learning algorithm (deep learning, convolutional neural network, recurrent neural network, etc.), hyperparameter settings (number of units in the fully connected layer, dropout layer ratio, learning rate, etc.), and filter size (n x n for a normal CNN, window size n for a one-dimensional CNN).

[0130] Next, the learning process procedure, including batch size, number of iterations, and number of epochs, is set (step S8-2), and a learning dataset consisting of pre-prepared training data on images and vegetation of the target area is loaded (step S8-3). Then, the training dataset is input to the learning model, the estimation result of the learning model is output, the error between the estimation result and the correct value (training data) is calculated, and the parameters of the learning model (CNN, etc.) are sequentially updated using methods such as backpropagation and gradient descent (steps S8-4 to S8-7). Finally, once the learning process is completed for all segments (step S8-8), the classifier generated by deep learning (the first learning model) is stored in the memory unit (step S8-9).

[0131] 2-2-5. Summary As described above, according to the present invention, a first trained model is generated that performs segmentation and classification processing of vegetation based on high-resolution UAV images, and estimation results regarding the vegetation ratio (vegetation cover rate) at satellite resolution can be output using this first trained model. Then, by using this estimation result as training data for the next stage second trained model, it becomes possible to generate training data for images taken by drones or UAVs or satellite images without performing time-consuming annotation.

[0132] To summarize, Figure 9 will explain the detailed processing flow of the inference phase, from UAV images without training data to outputting estimation results regarding satellite-resolution vegetation ratio (vegetation cover) using the first trained model. Figure 9 is a diagram showing the details of the processing flow corresponding to steps S5-6 to S5-12 in Figure 5. First, UAV images of the target area (including those without training data) are acquired from the UAV image DB of the memory unit (step S9-1). Then, preprocessing such as HSV hue conversion processing to convert the color space, or orthorectification processing to correct image distortion is performed (step S9-2).

[0133] Then, an HSV image converted to a color space suitable for vegetation determination is generated (step S9-3), and segmentation processing (SLIC, etc.) is performed to segment each vegetation (step S9-4). After the segmentation processing, the image is input to a classifier using a first trained model (a machine learning model such as RF or deep learning) to perform vegetation classification (step S9-5). The first trained model outputs the vegetation classification image of the target area and grid information (UAV resolution, e.g., 1 cm x 1 cm) (step S9-6).

[0134] This vegetation classification image is an estimation result using the first trained model, but sufficient accuracy can be ensured by preparing training data with a high resolution, such as 1 cm × 1 cm or 3 cm × 3 cm (see the explanation of Figure 19 below).

[0135] Next, grid information (satellite resolution) corresponding to the field plots of the target area, generated by the satellite-side processing, is acquired (step S9-7). For details on the process of generating the satellite-resolution grid information generated by the satellite-side processing, please refer to Figure 4. Then, referring to the grid information (satellite resolution) generated by the satellite-side processing, the high-resolution (UAV resolution) vegetation classification image (estimated value) generated by the first trained model on the drone side is converted to a lower resolution satellite image and divided into a grid to generate a vegetation ratio map (vegetation cover map) (step S9-8).

[0136] The generated vegetation ratio map (vegetation cover map) is stored in the memory unit (step S9-9), and a low-resolution vegetation ratio learning dataset is generated by associating low-resolution data of the target area photographed by satellite as input data and the vegetation ratio map (vegetation cover map) as training data (step S9-10), and then stored in the memory unit (step S9-11). Note that the processing in step S9-10 corresponds to the data merging process in step S3-1 in Figure 2.

[0137] Here, the significance of the processing in steps S9-8 (first invention) to S9-10 (second invention) lies, firstly, in that by converting the high-resolution (UAV resolution) vegetation classification image (estimated value) generated on the drone side into a low-resolution satellite image, it becomes possible to convert and represent information that shows one plant classification per pixel into the vegetation ratio (the percentage of the ground surface covered by a particular plant, also called vegetation cover) of at least one or more plant classifications per pixel.

[0138] Secondly, by generating training data for the next stage, the second trained model, from high-resolution (UAV resolution) vegetation classification images (estimated values) estimated using high-definition, sufficiently accurate training data, sufficient accuracy can be ensured even when performing two stages of estimation processing with the first and second trained models.

[0139] 2-3. Learning Process After Combining Drone and Satellite Data Next, the second learning process after combining data from the drone and satellite will be explained using Figure 10. Figure 10 is a diagram showing an example of the second learning process (learning phase) after data combination, and is a diagram showing the processing flow when implemented with a deep learning algorithm. First, a low-resolution satellite-resolution vegetation ratio learning dataset is obtained from the storage unit by combining satellite images (MS images with grid information) of the target area with vegetation ratio maps (vegetation cover maps) (low-resolution data at satellite resolution level) of at least one or more plant classifications generated based on high-resolution (UAV resolution) vegetation classification images (estimated values) generated on the drone side, in correspondence with satellite images (MS images with grid information) (step S10-1).

[0140] Next, a machine learning process (second learning process) is performed to determine the vegetation ratio from low-resolution satellite-resolution data captured by a satellite, using a low-resolution satellite-resolution vegetation ratio learning dataset (step S10-2). A predetermined number of datasets are prepared and appropriately divided into n subsets of a predetermined batch size, and n deep learning processes are performed. This completes one epoch, and the parameters of the neural network are updated as appropriate using methods such as backpropagation and gradient descent.

[0141] Then, the deep learning process is repeated for the set number of epochs. Once the learning process for all epochs is complete, the generation of a second trained model that estimates vegetation ratios from satellite images is completed (step S10-3). The classifier using the second trained model that estimates vegetation ratios from satellite images is stored in the memory unit (step S10-4).

[0142] Here, when using multispectral images (one-dimensional data) as input data for the deep learning model used in the learning process, it is preferable to use a one-dimensional CNN from the viewpoint of computational efficiency, etc. However, when processing multispectral images to convert them into two-dimensional data, a general two-dimensional CNN can also be used. The configuration of the one-dimensional CNN will be described later (see Figures 12 and 13). Note that the process in step S10-2 in Figure 10 is defined as the third invention, and the second learning process in step S10-3 is defined as the sixth invention.

[0143] 2-4. Operation using the second pre-trained model (inference phase) Next, the operation using the second pre-trained model (inference phase) will be explained using Figure 11. Figure 11 is a diagram showing an example of the processing flow of the inference phase using the second pre-trained model, and is a diagram showing an example of operation using the second pre-trained model after deep learning (second training process) (the entire vegetation discrimination system from satellite images).

[0144] First, satellite images (low-resolution satellite data) (MS image data with grid information) of the target area to be evaluated or analyzed are obtained from the memory unit (step S11-1). Here, the satellite images of the target area to be evaluated or analyzed may, of course, be unsupervised data, or data purchased from a satellite image provision site, etc. Then, a classifier using a second pre-trained model that estimates vegetation ratios from satellite images is loaded from the memory unit into the main memory, and the satellite images of the target area (low-resolution satellite data) are input into the classifier using the second pre-trained model (step S11-2).

[0145] Then, a classifier using the second trained model determines a predetermined vegetation ratio for each grid in the target area (vegetation discrimination processing) (step S11-3). In this step, the process of estimating the vegetation ratio for multiple plant classifications and bare ground for each pixel of the satellite image is called vegetation discrimination processing. The estimated results regarding the output vegetation ratios are stored in the estimation result DB of the memory unit (step S11-4). The estimated results regarding the vegetation ratios can be appropriately compared with topographic data, mapped to the evaluation target area, and displayed separately by color coding or other means for each vegetation classification (see Figures 20, 22, and 24).

[0146] 3. Learning Process When Using a One-Dimensional Convolutional Neural Network 3-1. Learning Process in Deep Learning for the Second Learning Process Next, the learning process when using a one-dimensional convolutional neural network (1D-CNN) in deep learning for the second learning process will be explained using Figures 12 and 13. Generally, CNNs are used to learn local features of multidimensional data such as image data. On the other hand, in the case of multispectral images (MS images) and time-series signals dealt with in this invention, since they are one-dimensional signals, a one-dimensional convolutional neural network (1D-CNN), which is different from a normal CNN, is suitable.

[0147] Of course, it is also possible to convert a one-dimensional signal into two-dimensional data such as a spectrum and process it as an image using a regular CNN, in which case it may be possible to extract features that also take surrounding pixels into consideration. However, considering the computational cost of converting a one-dimensional signal to a spectrum, we first evaluated a classifier generated using a one-dimensional convolutional neural network (1D-CNN) and obtained sufficient performance. Therefore, the configuration of the one-dimensional convolutional neural network (1D-CNN) will be described below. In addition to the one-dimensional convolutional neural network (1D-CNN), it is also possible to use 1D-CNN-LSTM, which combines LSTM (Long Short Term Memory) with the one-dimensional convolutional neural network (1D-CNN), to improve resistance to broadband noise.

[0148] While typical CNNs use two-dimensional filters to perform convolution, one-dimensional convolutional neural networks (1D-CNNs) use a one-dimensional convolutional filter to slide along the wavelength band direction, matching the one-dimensional data of the input signal, to extract the features of the pixel (features of a certain vegetation classification). As a result, a convolutional filter that emphasizes the characteristic wavelength band of the signal to be classified (MS image) can be obtained, effectively reflecting characteristics such as how a particular plant fits a certain reflection characteristic, and enabling appropriate vegetation classification.

[0149] Figure 12 is a diagram illustrating the overview of the learning phase, showing how input data is fed into a one-dimensional convolutional neural network (1D-CNN), and how the parameters of the learning model are updated based on the error with the training data using algorithms such as backpropagation and gradient descent. Figure 12 shows that MS image data (multispectral images) with grid information is prepared as input data, and vegetation ratio maps (vegetation cover maps) for plant classification are prepared as training data.

[0150] Here, the input data, MS image data (multispectral image), is an image captured by sensors in multiple wavelength bands (multiband), and consists of data acquired for each wavelength band (BAND) of each sensor (see Figures 16 and 17). In Figure 12, the structure of the input data is defined as consisting of B1, B2, B3, ... data, taking the initial letter "B" from "BAND," and the diagram illustrates that the 1st to nth data points are prepared for each pixel of the grid of the target area.

[0151] Furthermore, the input stage of the one-dimensional convolutional neural network (1D-CNN) is shown to be input in a single row, labeled B1, B2, B3, etc., corresponding to the structure of the input data. As for the training data, although this is just one example, it is shown that data is prepared with labels such as grass (GR), legume (LG), other weeds (MW), etc., assigned to each grid. Correspondingly, the output stage of the one-dimensional convolutional neural network (1D-CNN) is shown to have output items such as grass percentage, legume percentage, and other weed percentage, although this is just one example.

[0152] 3-2. Construction of a One-Dimensional Convolutional Neural Network (1D-CNN) As a model for deep learning in the second learning process, in addition to a regular CNN, a one-dimensional convolutional neural network (1D-CNN) can be used as an example. The structure when using this one-dimensional convolutional neural network (1D-CNN) will be explained using Figure 13. Figure 13(A) shows an example of the configuration image of a one-dimensional convolutional neural network (1D-CNN).

[0153] The input layer is shown as a sequence of inputs, B1, B2, B3, ... corresponding to the structure of the input data. The next stage, the convolutional layer, performs convolution processing with a filter to extract features. The following pooling layer is responsible for reducing the spatial resolution of the feature map by one level (downsampling) by aggregating the feature maps extracted in the convolutional layer into representative values ​​for each spatial locality. Similar convolution and pooling are performed in subsequent stages, a fully connected hidden layer is used, and the output layer outputs estimated results such as grass forage percentage, legume forage percentage, grass weed percentage, and other weed percentage.

[0154] Figure 13(B) shows an example of the configuration of a one-dimensional convolutional neural network (1D-CNN). From left to right in the table, the names and types of layers such as convolutional layers and pooling layers, and the set parameters (number of input channels, number of output channels, filter size (kernel size), stride length for moving the filter, and padding length for filling gaps in each layer) are defined.

[0155] 4. Process for Creating Vegetation Ratio Maps from Vegetation Classification Images 4-1. Overview of the Process for Creating Vegetation Ratio Maps from Vegetation Classification Images Next, we will explain the general content of the process for creating vegetation ratio maps (vegetation cover maps) from vegetation classification images using Figure 14. Figure 14 is a diagram of the process on the UAV side, showing the process of generating low-resolution vegetation ratio maps (vegetation cover maps) using the prediction results of a machine learning model (high-resolution vegetation classification images) obtained by the first learning process using high-resolution data of the UAV resolution. In the explanation of the processing flow described above, the content of the vegetation ratio map (vegetation cover map) generation process was briefly explained in steps S2-5 to S2-7 in Figure 2, steps S5-8 to S5-10 in Figure 5, and steps S9-6 to S9-9 in Figure 9, but here we will explain the specific processing content.

[0156] In practice, the process is performed using vegetation classification images at UAV resolution (for example, images where each pixel is 1 cm x 1 cm) (Step 1), but for the convenience of creating diagrams, we will explain using diagrams with reduced resolution vegetation classification images from UAV resolution (Step 2). The vegetation classification images are actually displayed in color, and each pixel contains multiple pieces of information about the vegetation. More specifically, for example, one pixel may contain information indicating that it is classified as pasture grass, information indicating that it is classified as weed, and information indicating that it is classified as bare ground.

[0157] Therefore, for example, it is possible to categorize each grid into areas classified as pasture, areas classified as weeds, and areas classified as bare ground, and represent these with vegetation classification images at UAV resolution, separated by vegetation type, as shown in step 3. For example, looking at the "areas classified as pasture" in step 3, if the ratio of pasture vegetation is calculated for a given 3x3 pixel area, areas with a high ratio of pasture vegetation will be colored black, areas with a medium ratio will be colored gray, and areas with a low ratio will be colored white.

[0158] This is illustrated in the diagram in Step 4. Looking at the areas classified as weeds, it can be seen that in the UAV-resolution vegetation classification image separated by vegetation, one plant classification is represented per pixel, whereas in the vegetation ratio map (vegetation cover map), each pixel represents the proportion of weeds (weed rate). Looking at the overall image, it can be seen that in the UAV-resolution vegetation classification image, areas where pixels classified as weeds are concentrated are black, areas where pixels classified as weeds are sparse are gray, and areas where there are no pixels classified as weeds are close to white.

[0159] 4-2. Details of the process for creating a vegetation ratio map from vegetation classification images Next, using Figure 15, we will explain in detail the process of creating a vegetation cover map (vegetation ratio map) from vegetation classification images, using pasture grass coverage as an example. In reality, the ratio within a pixel is calculated for a wider area than 3x3, but in Figure 15, for the sake of illustration and explanation, we show the process of calculating the vegetation ratio within a consolidated area of ​​3x3 pixels in a vegetation classification image.

[0160] More specifically, in practice, the proportion of each indicator (grass percentage, weed percentage, bare ground percentage, etc.) within a pixel at satellite resolution (tens of centimeters to tens of meters) is calculated based on the vegetation classification results (vegetation classification images) for each pixel at drone and UAV resolution (1 cm to tens of centimeters). This means that the proportion of each indicator will be calculated for hundreds to thousands of pixels.

[0161] First, in the vegetation classification image at UAV resolution (high resolution), pixels from the first 3x3 pixel region are extracted (Step 1). The percentage of pixels in the extracted 3x3 pixel region that are classified as pasture grass is calculated. In the figure, 8 out of 9 pixels are classified as pasture grass, so the calculation is 8 / 9 = approximately 89%. In this case, since the ratio is quite high at approximately 89%, a dark gray color is assigned to generate the first pixel of the vegetation ratio map (vegetation cover map) (Step 2).

[0162] By repeating the same process, the mth 3x3 pixel of the vegetation classification image is extracted (step m), and the percentage of pixels classified as pasture grass is calculated. Since 4 out of 9 pixels are classified as pasture grass, the calculation is 4 / 9 = approximately 44%. In this case, since 44% is a moderate percentage, gray is assigned to generate the mth pixel of the vegetation ratio map (vegetation cover map) (step m+1). Repeating the above process until the last 3x3 pixel area completes the vegetation ratio map (vegetation cover map) for pasture grass in the target area.

[0163] As described above, in vegetation classification images (high-resolution vegetation classification images), each pixel represents the classification result regarding grass species and vegetation, whereas in vegetation ratio maps (vegetation cover maps), instead of classifying each pixel as grass species or vegetation, the proportion within each pixel can be represented. Note that the above explanation focused on an indicator related to pasture coverage, but vegetation cover can be represented for multiple types of vegetation.

[0164] 5. Specific Examples of Multispectral Images (MS Images) 5-1. Example of Multispectral Image (MS Image) Composition Next, a specific example of a multispectral image (MS image) will be explained using Figure 16. Figures 16(A) and 16(B) show the contents of an MS image using Landsat 8 / 9 (7 bands, resolution 30 m) as an example. According to the left side of Figure 16(A), taking the i-th pixel of the satellite image as an example, the satellite image is composed of numerical values ​​acquired by the Band 1 sensor (reflectance from objects on the Earth's surface), numerical values ​​acquired by the Band 2 sensor, ... numerical values ​​acquired by the Band 7 sensor among sensors of multiple wavelength bands.

[0165] In the case of Landsat 8 / 9 (7 bands, 30 m resolution), the bands consist of band 1 (Coastal / Aerosol, wavelength range 433-453 nm), band 2 (Blue, wavelength range 450-515 nm), band 3 (Green, wavelength range 525-600 nm), band 4 (Red, wavelength range 630-680 nm), band 5 (NIR, wavelength range 845-885 nm), band 6 (SWIR1, wavelength range 1560-1660 nm), band 7 (SWIR2, wavelength range 2100-2300 nm), etc. This is because each wavelength band is better suited to detecting different colors and substances, so it is desirable to use sensor data from multiple wavelength bands (multispectral or hyperspectral).

[0166] Furthermore, as shown in the right-hand diagram of Figure 16(A), the values ​​acquired by the sensors for each band are plotted by frequency, and the values ​​for band 1, band 2, band 3, ..., band 7 are shown in a line graph from left to right. Here, the gray areas (bundles of lines) are superimposed displays of the reflectance of each pixel (each grid) from the first to the second, and the solid line shows the average reflectance of all pixels (all grids). Figure 16(B) is a diagram showing the structure of multispectral image (MS image) data, showing that the MS image data is composed of bundles of MS images of each pixel from the first to the nth pixel.

[0167] 5-2. Specific Examples of MS Image Data from Major Optical Satellites Next, specific examples of MS image data from major optical satellites will be explained using Figure 17. The top row shows an example of MS image data acquired from the Sentinel-2 satellite. The Sentinel-2 MS image data consists of bundles of 12-band, 10m resolution MS images, and as with Figure 16, it shows that the MS image data is composed of bundles of MS images for each pixel from the 1st to the nth pixel. The middle row shows MS image data from Landsat 8 / 9 (7-band, 30m resolution), which was explained in Figure 16, so the details will be omitted. The bottom row shows MS image data acquired from the PlanetScope satellite, and it can be seen that it consists of bundles of 8-band, 3m resolution MS images.

[0168] 5-3. Example of MS Image Data in Tabular Format Next, using the Landsat 8 / 9 (7 bands, 30m resolution) as an example, we will explain an example of how MS image data can be represented in tabular format using Figure 18. Figure 18 is a diagram showing an example of the table structure of satellite data in the case of Landsat 8 / 9 (7 bands, 30m resolution). In Figure 18, FID indicates the ID of each grid (pixel in the image), and it can be seen that the reflectance acquired by the sensor is recorded for each band of multiple wavelength bands. In Figures 16 and 17, multispectral images (MS images) were represented by explaining them in graph format, but in terms of data processing, it is desirable to save them in such a table format.

[0169] 6. Accuracy of Estimation Results Output by the First Trained Model in Drone-Side Processing When using the estimation results output by the first trained model in drone-side processing as training data for the next stage, the second training process, we verified the accuracy of this training data, which will be explained using Figure 19. Figure 19 is a diagram showing an example of performance evaluation (confusion matrix) of training data, which is the estimation result by the first trained model generated from images taken by a drone or UAV. For the performance evaluation of the first trained model, five classifications were prepared as classification targets: grasses, "Grass family grasses (GR)" and "Leguminous family grasses (LG)" as pasture grasses; grasses, "Grass family weeds (GW)" and "Other weeds (MW)" as weeds; and "Bare ground / dead leaves (BG)" as everything else.

[0170] As also shown in Figure 19, the details of the vegetation indicated by each symbol are as follows: <Details of the vegetation indicated by each symbol> GR: Timothy, Orchardgrass LG: Alfalfa, White clover GW: Reed canarygrass, Sooty grass, Redtop, Kentucky bluegrass, Meadow foxtail, Cabbage, Sparrow's hornbeam MW: Dandelion, Dock, Shepherd's purse, Giant horseweed, Wild rose, Butterbur

[0171] For this performance evaluation, data with a GSD size of 1 cm x 1 cm was used. The target area was 4 areas, 9 regions, and 41 plots of land. Of these, Sentinel-2 satellite images were obtained for 28 plots in 8 regions across 4 areas. Correspondingly, the parameters for grass species discrimination processing of UAV images (segment size = 5 pixels) were determined, and training data for 14 plots in 8 regions was generated using the second pre-trained model. As shown in Figure 19, the accuracy rate when the entire dataset was divided into the five categories mentioned above was 88%, indicating sufficient accuracy.

[0172] Here, precision refers to the percentage of predictions that were actually positive, while recall refers to the percentage of predictions that were actually positive. The F1 score is an index that emphasizes the balance between precision and recall, and specifically refers to the harmonic mean of precision and recall. Both indices are between 0.87 and 0.88, indicating that sufficient accuracy has been achieved.

[0173] In this invention, the prediction results of the first trained model are used as training data for the second training process (after being converted into a satellite-resolution vegetation ratio map), so sufficient accuracy is required. According to this evaluation result, sufficient accuracy is ensured, and it is deemed necessary and sufficient as training data for the second training process. Note that this result was obtained using data with a GSD size of 1 cm x 1 cm, but even if data of, for example, 3 x 3 cm is used, the necessary accuracy can be ensured, although the accuracy will be slightly reduced, so the GSD size can be selected as appropriate.

[0174] 7. Performance Evaluation of the Second Trained Model (Performance Evaluation of the Entire System) Next, the performance evaluation results of the learner using the second trained model will be explained using Figures 20 to 26. Note that the estimation process by the learner using the second trained model is a system-wide process that takes satellite images as input and outputs vegetation ratio maps (vegetation cover maps) such as vegetation cover rate as estimation results. Therefore, the performance evaluation of the learner using the second trained model is also an evaluation of the performance of the entire system.

[0175] 7-1. Using images from the Sentinel-2 satellite 7-1-1. Quantitative evaluation Figure 20 shows the performance evaluation results of the second trained model when using images from the Sentinel-2 satellite. Figure 20(A) shows an example of the mean and standard deviation of R2 and RMSE after repeating k-fold cross-validation (k=5) 10 times with the second trained model. (In the case of the Training dataset (n=5,148)) Here, k-fold Cross Validation is a validation method in which the dataset is divided into k groups, one of the k groups is used as test data, and the remaining data is used as training data.

[0176] The dataset (n=5,148) means that data from 5,148 pixels of Sentinel-2 satellite images (1 pixel = 10m x 10m) was used. In the case of a field size of 200m x 200m (400 pixels per field), this corresponds to data from approximately 12 fields. Figure 20(B) shows the relationship between measured and predicted values ​​of the Test data that was not used for model training. (Test dataset (n=1,288)) Note that multiple datasets are prepared and used separately as a Training dataset for the learning phase and a Test dataset for validation in the inference phase.

[0177] Here, R² (R squared) and RMSE are metrics used to compare methods and models. R² represents the proportion of the variability (variance) of the dependent variable that the model was able to explain. For example, if R² is 0.8, it means that 80% of the variability of the dependent variable was explained by the model's predicted values. RMSE stands for root mean squared error and is a measure frequently used as the difference between the value predicted by the model or estimator (sample value or population value) and the observed value. A value closer to 0 indicates that the model and data fit well.

[0178] As shown in Figure 20(A), R2 is at a high level of approximately 0.62 to 0.80, and RMSE is within a low range of 0.036 to 0.113, indicating that the prediction results of the second trained model are sufficiently accurate.

[0179] Furthermore, as shown in Figure 20(B), R2 is at a high level of approximately 0.53 to 0.75, and RMSE is within a low range of 0.05 to 0.14, indicating that the prediction results of the second trained model are sufficiently accurate. <Performance evaluation results in Figure 20(B)> Prediction of vegetation cover (VC): R2 (0.75), RMSE (0.05) Prediction of grass cover (GR): R2 (0.74), RMSE (0.14) Prediction of legume cover (LG): R2 (0.68), RMSE (0.12) Prediction of grass weed cover (GW): R2 (0.73), RMSE (0.06) Prediction of other weed cover (MW): R2 (0.53), RMSE (0.07)

[0180] 7-1-2. Qualitative Evaluation Figure 21 is an example of the performance evaluation results of a vegetation ratio map (vegetation cover map) of a second pre-trained model using drone (measured values) and satellite data (predicted values) when using images from the Sentinel-2 satellite. The example shows a field in Obihiro City (here referred to as the OBH_M7C field (276 data points)). The upper part of Figure 21 is a vegetation ratio map (measured values) generated by adding plant classification information obtained from field surveys to images taken by a drone in a predetermined target area. The lower part is the output result (predicted values ​​of the vegetation ratio map) estimated by the learner of the second pre-trained model based on images taken by satellite in a predetermined target area.

[0181] In Figure 21, GR represents grasses, LG represents legumes, and MW represents other weeds. The closer to black the color, the higher the vegetation cover of the plant (a value of 1.0 in the upper and lower bar graphs means 100% vegetation cover, and 0.0 means 0% vegetation cover), and the closer to white the color, the lower the vegetation cover. In Figure 21, although the evaluation is based on visual observation, it can be seen that the measured values ​​in the upper section and the prediction results of the classifier using the second trained model in the lower section match to a considerable extent.

[0182] This result clearly demonstrates the results of the quantitative evaluation in 7-1-1. This is presumably because the Sentinel-2 satellite covers a wide range of wavelengths from 443 nm to 2190 nm, consisting of a total of 12 wavelength bands (see Figure 17), which allows the learning model to extract the characteristics of plant reflection properties.

[0183] 7-2. Using Landsat satellite imagery 7-2-1. Quantitative evaluation

[0184] Figure 22 shows the performance evaluation results of the estimation results by the second trained model when using Landsat satellite images. Figure 22(A) shows an example of the mean and standard deviation of R2 and RMSE after repeating k-fold cross-validation (k=5) 10 times with the second trained model. (Training dataset (n=436)) Here, the meaning of the dataset (n=436) is that data from 436 pixels of Landsat satellite images (1 pixel is 30m x 30m) was used, and in the case of a field size of 200m x 200m (approximately 36 pixels per field), this corresponds to data from approximately 12 fields.

[0185] Figure 22(B) shows the relationship between the measured and predicted values ​​of the Test data that was not used for model training. (Test dataset (n=109)) Note that a large number of datasets are prepared and used separately as a Training dataset for the training phase and a Test dataset for validation in the inference phase. According to Figure 22(A), the R2 is at a high level of approximately 0.68 to 0.84, and the RMSE is in a low range of 0.023 to 0.091, indicating that the prediction results of the second trained model have sufficient accuracy.

[0186] Furthermore, as shown in Figure 22(B), R2 is at a high level of approximately 0.63 to 0.86, and RMSE is within a low range of 0.03 to 0.13, indicating that the prediction results of the second trained model are sufficiently accurate. <Performance evaluation results in Figure 22(B)> Prediction of vegetation cover (VC): R2 (0.63), RMSE (0.03) Prediction of grass cover (GR): R2 (0.72), RMSE (0.13) Prediction of legume cover (LG): R2 (0.60), RMSE (0.12) Prediction of grass weed cover (GW): R2 (0.86), RMSE (0.07) Prediction of other weed cover (MW): R2 (0.70), RMSE (0.04)

[0187] 7-2-2. Qualitative Evaluation Figure 23 is an example of the performance evaluation results of a vegetation ratio map (vegetation cover map) of a second pre-trained model using drone (measured values) and satellite data (predicted values) when using Landsat satellite images. The example shows a field in Obihiro City (here referred to as OBH_M7C field (22 data points)). The upper part of Figure 23 is a vegetation ratio map (measured values) generated by adding plant classification information obtained from field surveys to images taken by a drone in a predetermined target area. The lower part is the output result (predicted values ​​of the vegetation ratio map) estimated by the learner of the second pre-trained model based on images taken by satellite in a predetermined target area.

[0188] In Figure 23, GR represents grasses, LG represents legumes, and MW represents other weeds. The closer to black the color, the higher the vegetation cover of the plant (a value of 1.0 in the upper and lower bar graphs means 100% vegetation cover, and 0.0 means 0% vegetation cover), and the closer to white the color, the lower the vegetation cover. In Figure 22, although the evaluation is based on visual observation, it can be seen that the measured values ​​in the upper panel and the prediction results of the classifier using the second trained model in the lower panel match to a considerable extent.

[0189] This result clearly demonstrates the results of the quantitative evaluation in 7-2-1. This is presumably because the Landsat 8 / 9 satellites cover a wide range of wavelengths from 433 nm to 2200 nm, consisting of a total of seven wavelength bands (see Figure 17), which allows the learning model to extract the characteristics of plant reflection properties.

[0190] However, compared to the Sentinel-2 satellite, the Landsat 8 / 9 satellites appear to be slightly inferior in both quantitative and qualitative evaluations. This is likely because the Sentinel-2 satellite covers a wide wavelength range from 443 nm to 2190 nm, consisting of 12 wavelength bands in total, whereas the Landsat 8 / 9 satellites, while having a wide wavelength range, have fewer bands (7). Therefore, from this comparison of the two satellite data, it can be inferred that further improvements in accuracy can be expected by widening the multiband bandwidth and increasing the number of bands.

[0191] 7-3. When using images from PlanetScope satellites 7-3-1. Quantitative evaluation

[0192] Figure 24 shows the performance evaluation results of the estimation results by the second trained model when using images from the PlanetScope satellite. Figure 24(A) shows an example of the mean and standard deviation of R2 and RMSE after repeating k-fold cross-validation (k=5) 10 times with the second trained model. (Training dataset (n=52,055)) Here, the meaning of the dataset (n=52,055) is that data equivalent to 52,055 pixels of images from the PlanetScope satellite (1 pixel is 3m x 3m) was used, and in the case of a field size of 200m x 200m (approximately 4356 pixels per field), this corresponds to data for approximately 12 fields.

[0193] Figure 24(B) shows the relationship between the measured and predicted values ​​of the Test data that was not used for model training. (Test dataset (n=13,014)) Note that a large number of datasets are prepared and used separately as a Training dataset for the training phase and a Test dataset for validation in the inference phase. According to Figure 24(A), although R2 is at a slightly low level of around 0.274 to 0.549, the RMSE is within a low range of 0.066 to 0.19, indicating that the prediction results of the second trained model have sufficient accuracy.

[0194] Furthermore, as shown in Figure 24(B), although R2 is at a slightly low level of approximately 0.43 to 0.56, RMSE is within a low range of 0.07 to 0.19, indicating that the prediction results of the second trained model are sufficiently accurate. <Performance evaluation results in Figure 24(B)> Prediction of vegetation cover (VC): R2 (0.53), RMSE (0.07) Prediction of grass cover (GR): R2 (0.56), RMSE (0.19) Prediction of legume cover (LG): R2 (0.43), RMSE (0.17) Prediction of grass weed cover (GW): R2 (0.44), RMSE (0.10) Prediction of other weed cover (MW): R2 (0.46), RMSE (0.09)

[0195] 7-3-2. Qualitative Evaluation Figure 25 is an example of the performance evaluation results of a vegetation ratio map (vegetation cover map) of a second pre-trained model using drone (measured values) and satellite data (predicted values) when using images from the PlanetScope satellite. The example shows a field in Obihiro City (here referred to as OBH_M7C field (3,380 data points)). The upper part of Figure 25 is a vegetation ratio map (measured values) generated by adding plant classification information obtained from field surveys to images taken by a drone in a predetermined target area. The lower part is the output result (predicted values ​​of the vegetation ratio map) estimated by the learner of the second pre-trained model based on images taken by satellite in a predetermined target area.

[0196] In Figure 25, GR represents grasses, LG represents legumes, and MW represents other weeds. The closer to black the color, the higher the vegetation cover of the plant (a value of 1.0 in the upper and lower bar graphs means 100% vegetation cover, and 0.0 means 0% vegetation cover), and the closer to white the color, the lower the vegetation cover. In Figure 25, although the evaluation is based on visual observation, it can be seen that the measured values ​​in the upper section and the prediction results of the classifier using the second trained model in the lower section do not match at a fine level, but the general trend matches to a certain extent.

[0197] However, compared to the Sentinel-2 and Landsat 8 / 9 satellites, the evaluation is considered to be slightly inferior in both quantitative and qualitative aspects. This is presumably because the PlanetScope satellite only covers a relatively narrow wavelength range from 455 nm to 860 nm, consisting of a total of eight wavelength bands (see Figure 17), which prevents the learning model from extracting the characteristics of plant reflections very well. Therefore, from the comparison of the three satellite data above, it can be inferred that further improvements in accuracy can be expected by widening the multiband bandwidth and increasing the number of bands.

[0198] 8. System Configuration Next, an example of the system configuration of the vegetation discrimination system 800 of the present invention will be described using Figure 26. The vegetation discrimination system 800 consists of a server (including a cloud) or external device 300, a terminal 400, a network or communication means 500, a drone or UAV, a satellite system 700, and the like.

[0199] 8-1. Drone or UAV600 UAV images are acquired by flying a drone or UAV600 at a predetermined altitude and speed. For shooting, a 2K, 4K, or 8K equivalent CCD camera is used, and the altitude and flight speed are adjusted to appropriately adjust the resolution when converted to the ground surface (e.g., 1cm x 1cm, 3cm x 3cm). Since the images are taken while moving over objects on the ground, information on the height of the objects on the ground can also be acquired. Based on the captured images, 3D data and orthomosaic images corrected for distortion to match the field are generated by matching them with geographic information.

[0200] In this invention, the UAV image is an image used to generate a first trained model, and since the vegetation ratio map output by the first trained model is used as training data for the subsequent second training process, the UAV image only needs to be an image taken at a predetermined resolution, and may be, for example, an image taken with an RGB camera. (Note that the second trained model uses multispectral satellite imagery.)

[0201] More specifically, the UAV images are intended for use in assigning plant classifications such as grasses (GR), legumes (LG), etc., at a predetermined resolution, and images captured with an RGB camera are sufficient. Furthermore, it is considered possible to improve the accuracy of the first trained model by using a multispectral camera, including a near-infrared camera, in addition to an RGB camera, for the UAV images. The captured UAV images are stored on a server (including the cloud) or external device 300 as appropriate via a network or communication means 500.

[0202] 8-2. Satellite System 700 Satellite images are used for generating (training phase) and operating (inference phase) the second trained model. Satellite images are broadly divided into multispectral images and hyperspectral images. Examples of multispectral images include images from the Sentinel-2 satellite (12 bands), Landsat 8 / 9 satellite (7 bands), and PlanetScope satellite (8 bands). Examples of superspectral images include images from the HISUI satellite (185 bands). Satellite images can be obtained from various commercial databases, and satellite images obtained via the network or communication means 500 are stored on a server (including the cloud) or external device 300 as appropriate.

[0203] 8-3. Server (including cloud) or external device 300, terminal 400 The server (including cloud) or external device 300 (hereinafter abbreviated as "server, etc.") or terminal 400 is a device that performs the processing of each flowchart described above (Figures 4 to 11) and various data processing (Figures 12 to 25). Either the terminal or the server, etc., or both, are equipped with a control unit, a storage unit, and a communication unit (see Figure 27), and the vegetation identification processing is performed by each device through a division of roles, such as data input and screen display of processing results via the input unit and display unit of the terminal.

[0204] The server, etc., can be a server or cloud, or any computer with a certain level of processing power, such as a desktop PC or laptop computer. Although not shown in the figures, the server, etc. 300 or terminal 400 includes control means consisting of a central processing unit (CPU) for processing data and performing various operations, storage means for storing programs and various data and temporarily storing the progress of processing, input means for inputting various setting values, output means for printing or displaying judgment results, and communication means for communicating with a communication network (see Figure 27).

[0205] 8-4. Network or communication means 500 The network or communication means 500 is a wired or wireless communication line, and consists of, for example, a WAN (Wide Area Network) line such as the Internet, a LAN (Local Area Network) line such as a wireless LAN, and a wireless line for communicating with satellites or drones.

[0206] 9. Functional Block Diagram Next, an example of a functional block diagram of the server (including the cloud), external device 300, or terminal 400 of the vegetation discrimination system 800 shown in Figure 26 will be explained using Figure 27. For the sake of explanation, the explanation will be based on the example where the control unit and memory unit, etc., for executing the processing of each flowchart (Figures 4 to 11) and various data processing (Figures 12 to 25) described above are implemented in the server (including the cloud) or external device 300, and only the input and display processing is performed on the terminal side. However, the terminal 400 may perform all processing, or the server (including the cloud) or external device 300 and the terminal 400 may share the processing responsibilities.

[0207] The control unit is implemented on hardware by the CPU (Central Processing Unit) loading a program containing the processing steps of each flowchart (Figures 4 to 11) stored in the memory unit into temporary memory and executing each step. As shown in Figure 27, the control unit consists of a management unit that manages data (UAV data management unit, satellite data management unit, map information management unit, etc.), a learning processing unit that performs learning processing (first learning processing unit, second learning processing unit, first trained model, second trained model, low-resolution vegetation ratio learning data generation unit), and a map generation unit that generates classification images and vegetation ratio maps (vegetation cover maps) (high-resolution vegetation classification image generation unit, vegetation ratio map generation unit).

[0208] The first learning processing unit performs machine learning processing such as SLIC-RF or deep learning based on UAV images with training data to generate a first trained model. The high-resolution vegetation classification image generation unit uses the first trained model to generate high-resolution vegetation classification images (UAV resolution) related to vegetation classification based on high-resolution UAV images without training data.

[0209] The vegetation ratio map generation unit applies grid information at satellite resolution (low resolution) generated by satellite-side processing to high-resolution vegetation classification images (UAV resolution) to generate a satellite-resolution vegetation ratio map (vegetation cover map).

[0210] The low-resolution vegetation ratio learning data generation unit combines low-resolution data of the target area captured by satellite with a vegetation ratio map (vegetation cover map) of the target area generated by the vegetation ratio map generation unit, using them as training data, to generate a low-resolution vegetation ratio learning dataset.

[0211] The second learning processing unit uses the vegetation ratio map (vegetation cover map) generated by the drone's processing as training data, and performs machine learning processing using gridded satellite images of field plots generated by the satellite's processing as input data to generate a second trained model.

[0212] Note that the first and second trained models in the learning processing unit actually consist of structural information of the trained model and parameters of the trained model that have been updated during the learning process, which are stored in the memory unit. However, for the sake of explanation, they are described within the learning processing unit.

[0213] The memory unit contains a UAV image database that stores information and data necessary for processing on the drone side, and a satellite image database that stores information and data necessary for processing on the satellite side. The UAV image database stores UAV image data (with the corresponding training data stored if training data is available), segment data of the target area after segmentation processing, grid information of the target area (UAV resolution) generated based on field plot data, classification images output by the first learning model, and vegetation ratio maps generated from the classification images.

[0214] The satellite image database stores original satellite image data, MS image data (in table format or image information) generated from the satellite image data, field plot data, grid information (satellite resolution) for the target area generated from the field plot data, MS image data with grid information, and an estimation results database that stores estimation results regarding the vegetation ratio of the target area, output using a second trained model.

[0215] The terminal includes a display unit for showing the estimation results of the trained model, a control unit, a storage unit, a communication unit, and an input unit for configuring the training model for the training process and executing various commands.

[0216] 10. Economic Effects 10-1. Overview By using 10m resolution Sentinel-2 satellite images or 30m resolution Landsat-8 satellite images, it becomes possible to provide distribution maps of weed percentages and leguminous grass percentages within fields, as well as the long-term decline of pasture grasses. Therefore, by incorporating this invention as a "vegetation discrimination service using satellite images" into farming support services and utilizing it to determine the necessity of grassland renewal based on weed percentages and leguminous grass percentages, and to manage fertilization appropriately, it is expected to make a significant contribution to improving feed production.

[0217] 10-2. Marketability The Ministry of Agriculture, Forestry and Fisheries has set forth the goal of improving feed self-sufficiency as part of establishing livestock farming based on domestically produced feed. For example, the estimated self-sufficiency rate for all feed is 26% in FY2022, with a target of increasing it to 34% in FY2030, and for roughage, the self-sufficiency rate is 78% in FY2022, with a target of increasing it to 100% in FY2030. The technology of the present invention is a groundbreaking technology that allows for easy determination of the necessity of grassland renewal and the need for fertilization, and can be said to be a necessary technology for realizing national policies. In rural areas, there is a desire for the practical application of vegetation discrimination using satellite imagery that can be applied to large grasslands, and there are reports that grassland renewal can lead to a 35% increase in pasture yield, so the use of the present invention is expected.

[0218] 10-3. Economic Effects As a forecast of the industrial scale of pasture diagnostic services using the vegetation discrimination system of the present invention, for example, if the service fee is set at 400 yen per hectare, the service fee for all of Hokkaido (approximately 400,000 hectares) can be estimated at 160 million yen. Furthermore, by using the vegetation discrimination system of the present invention, it is expected that the proportion of pasture can be improved by enabling rapid treatment of weeds.

[0219] Therefore, assuming an increase of 7 to 8 percentage points in the proportion of pasture grass, the income increase for individual farmers can be estimated at 27,000 to 35,000 yen per livestock, and considering the total number of livestock in Hokkaido, this can be estimated at 12.4 to 16.1 billion yen per year. This value represents a 2.5 to 3.2% increase compared to the agricultural output of Hokkaido's dairy farming, which was 502.6 billion yen (FY2019), and can be considered a realistic figure.

[0220] 11. Summary of Features, Significance, and Effects of the Invention As described above, the present invention has the following features. Firstly, since multiple diverse vegetation types are distributed in an area of ​​one pixel of satellite resolution (on the order of several tens of centimeters to several tens of meters), it is possible to represent the distribution of diverse vegetation types by expressing this as the vegetation cover rate of multiple plant classifications.

[0221] Secondly, a key feature of this approach is that the training data input to the second pre-trained model utilizes the output results of the first pre-trained model. Therefore, the satellite-resolution vegetation ratio map (vegetation cover map) is based on estimated values ​​(vegetation classification images) output using the classifier of the first pre-trained model, making it possible to generate it without significant effort using UAV images that do not have training data attached.

[0222] Thirdly, according to the present invention, when generating the first trained model, high-resolution UAV images are used and high-resolution training data on vegetation classification obtained from meticulous field surveys is provided to ensure the accuracy of the estimation results of the first trained model. Furthermore, by converting the input to the next stage, the second trained model, into vegetation cover, which is the ratio of a predetermined plant covering the ground surface or the ratio of bare ground per pixel, it becomes possible to ensure the accuracy of the final estimation results after passing through two stages of trained models.

[0223] In this regard, conventionally, inputting the estimation results of the first pre-trained model into the subsequent second pre-trained model has generally been avoided because it involves layering estimations upon estimations, making it difficult to ensure the accuracy of the final estimation results. This can be seen as a technical obstacle that generally tends to lead to avoidance. Therefore, it can be said that this has been achieved by devising a method for generating the first pre-trained model and improving the accuracy of the estimation results of the first pre-trained model.

[0224] Fourthly, looking at the present invention from an overview perspective, it is characterized by its ability to perform vegetation discrimination in a way that solves both the problems of drones, such as the data processing capability issues due to the high resolution of UAV images and the decrease in prediction accuracy when the lighting conditions of the images differ, and the problems of satellites, which, although the image acquisition conditions are constant and wide-area photography is possible, have difficulty in distinguishing fine vegetation due to their low resolution.

[0225] Fifthly, conventionally, when performing machine learning such as deep learning using satellite images as they are, one plant classification was assigned to each pixel. However, in reality, it is difficult to manually calculate the vegetation ratio and assign training data for each pixel, which is actually several meters to tens of meters in size at satellite resolution and contains numerous vegetation classifications. The technology of the present invention is characterized by the fact that it enables vegetation discrimination on satellite images without annotating the satellite images themselves, by using processing such as annotation in segmentation processing and classification processing on the high-resolution images on the drone side as an intermediary.

[0226] Sixth, extending the present invention, a key feature is that by using the output (estimation result) of a first pre-trained model using high-resolution images as training data for a second training process using low-resolution images in the next stage, a second pre-trained model can be generated without annotating low-resolution images that are relatively easy to obtain but cover a wide area. This makes it possible to obtain high-resolution images by flying a drone over a relatively small area, for example, around 10 fields or a few farms or a single municipality, and then conduct field surveys to add training data for vegetation classification, generate a highly accurate first pre-trained model to generate training data for the second training, and then generate a second pre-trained model by linking it with readily available satellite images of the target area.

[0227] More specifically, and this is just one example, (1) In the learning phase of the first learning process, it is necessary to conduct field surveys and perform annotation, and considering the time and effort involved, the process is carried out using "supervised UAV images" from about 10 fields or several farms or one municipality. (2) In the inference phase using the first trained model, training data is not required, and field surveys and annotation are not necessary. Only UAV images obtained by flying a drone or UAV are needed. Therefore, "unsupervised UAV images" covering an area encompassing multiple municipalities (for example, the area of ​​the Tokachi General Subprefectural Bureau in Hokkaido) are input into the first trained model, and the output result from the first trained model (vegetation ratio map) is used as training data for the learning phase of the next stage, the second learning process. (3) In the next learning phase, the second learning process is carried out using the training data generated in (2) above and satellite images obtained from commercial databases, etc., for the same target area to generate the second trained model. (4) In the inference phase (operational stage) of the generated second trained model, it will also be possible to input satellite images of prefecture-level areas obtained from commercial databases, etc., and perform vegetation discrimination services at the Hokkaido or nationwide level.

[0228] In this way, by flying a drone in a relatively narrow area to acquire UAV images and generating training data through on-site surveys, a first trained model can be created. After this first trained model is generated, the output of the first trained model can be used as training data for the next stage, the second training process, without conducting further on-site surveys. After generating the second trained model, satellite images of the entire Hokkaido region can be purchased from commercial databases and applied to the second trained model to provide a vegetation discrimination service covering a wide area, such as the entire Hokkaido region. Furthermore, since the vegetation discrimination system of this invention utilizes the reflectance spectrum of plants, it can be universally applied to any region for similar plants, making it possible to analyze vast areas on a prefectural level for vegetation discrimination.

[0229] As described above, the classification or discrimination process of vegetation, such as grasses (GR), legumes (LG), grass weeds (GW), other weeds (MW), bare ground (BG), and vegetation cover (VC), has been explained as an example. The technology of the present invention can also be used for discrimination processing of various classifications related to plants, such as deciduous broad-leaved forests, evergreen coniferous forests, shrub communities, herbaceous plant communities, cultivated land, and non-vegetated areas, or classifications such as cedar, cypress, bamboo groves, pine, broad-leaved trees, water areas, shrubs and others, or trees (tall trees and shrubs), community height and vegetation height, biomass (vegetation cover × vegetation height), plant growth state, long-grass type, and short-grass type.

[0230] Furthermore, beyond classification processing related to plant classification, it can also be used for classification processing based on various information related to the attributes of the target area, such as land use status including forests, wasteland, rice paddies, fields and other agricultural land, pastures, fallow land, land under construction, vacant lots, industrial land, general low-rise residential areas, densely populated low-rise residential areas, medium- and high-rise residential areas, commercial and business land, road land, parks and green spaces, other public and public-interest facility land, rivers and lakes, disaster countermeasures such as the distribution of forests and landslides, and other attributes of the target area.

[0231] Furthermore, by utilizing the results of vegetation classification and attribute classification, it becomes possible to select fields for grassland development, determine fertilizer application rates according to vegetation type, set target yields, and formulate pasture harvesting plans. More specifically, for example, by using the estimated percentage of legumes, the amount of fertilizer to be applied can be determined according to the distribution of legumes, and the amount of fertilizer to be applied can be determined based on that (the same applies when using compost). Alternatively, although it is known that the optimal harvesting time differs depending on the grass species and variety, the predicted vegetation ratio can be used to formulate pasture harvesting plans.

[0232] Furthermore, by combining this with satellite imagery archives, it becomes possible to understand the changes in pasture conditions over time, which can be used to determine the necessity of grassland renewal.

[0233] 200 Non-temporary tangible recording media 300 Servers (including cloud) or external devices 400 Terminals 500 Networks or communication means 600 Drones or UAVs 700 Satellite systems 800 Vegetation discrimination systems

Claims

1. An information processing system for generating a learning dataset, which uses the estimation result, which is the output of a trained model generated by machine learning using high-resolution data of a predetermined target area photographed by a drone or UAV, as training data, and low-resolution data of a predetermined target area photographed by a satellite as input data, comprising: a high-resolution vegetation classification learning dataset generation unit that generates a high-resolution vegetation classification learning dataset by associating high-resolution data of a target area photographed by a drone or UAV with training data indicating the vegetation classification of the target area; a first learning processing unit that uses the generated high-resolution vegetation classification learning dataset to perform machine learning processing to determine the vegetation classification from the high-resolution data of a target area photographed by a drone or UAV; and a high-resolution vegetation classification image generation unit that uses the first trained model generated by the first learning processing unit to estimate the vegetation classification of the target area and output a high-resolution vegetation classification image. An information processing system characterized by comprising: a vegetation ratio map generation unit that generates a vegetation ratio map, which is a map showing the vegetation ratio, which is the proportion of the ground surface covered by a predetermined plant or the proportion of bare ground, based on the high-resolution vegetation classification image generated, and which is a vegetation ratio map obtained by converting the low-resolution data captured by the satellite to a predetermined low resolution according to the resolution and dividing it into a grid.

2. An information processing system for generating training data, characterized in that it comprises a low-resolution vegetation ratio training data generation unit, which generates a low-resolution vegetation ratio training dataset, in which a vegetation ratio map of the target area generated by the vegetation ratio map generation unit is associated with low-resolution data of the target area photographed by the satellite as training data.

3. An information processing system for learning, further comprising: a second learning processing unit that performs machine learning processing to determine the vegetation ratio from low-resolution data captured by a satellite, using a low-resolution vegetation ratio learning dataset obtained by associating the vegetation ratio map of the target area generated by the vegetation ratio map generation unit as training data with the low-resolution data of the target area captured by the satellite.

4. The information processing system according to claim 3, further comprising: a classifier that inputs the low-resolution vegetation ratio learning dataset, which is obtained by associating the vegetation ratio map of the target area generated by the vegetation ratio map generation unit as training data with the low-resolution data of the target area captured by the satellite, and performs machine learning processing on the second learning processing unit to generate a second trained model; and a classifier that determines the vegetation ratio of the target area based on the output from the second trained model.

5. An information processing system according to claims 1 to 4, characterized in that, in the vegetation ratio map which is training data generated by the vegetation ratio map generation unit, the label value indicating the correct answer is represented by a vegetation ratio that includes one or more of the following: the ratio of plants covering the ground surface, the grass ratio (or grass ratio, legume ratio), the grass ratio (or grass ratio, grass weed ratio), the legume ratio (or legume ratio, legume weed ratio), the weed ratio (or grass weed ratio, legume weed ratio, other weed ratio), and the bare ground ratio.

6. A trained model for a computer to determine the vegetation of a target area based on low-resolution data captured by a satellite, wherein, in the information processing system according to claim 3, the second training unit performs machine learning processing using a training dataset in which the vegetation ratio map of the target area generated by the vegetation ratio map generation unit is associated with the low-resolution data of the target area captured by the satellite as training data, and the second training unit performs machine learning processing to generate the second trained model.

7. The second trained model of claim 6, characterized in that the output of the second trained model is expressed as a vegetation ratio that includes one or more of the following: the percentage of the ground surface covered by plants, the percentage of pasture grass (or the percentage of grass pasture grass, the percentage of leguminous grass), the percentage of grasses (or the percentage of grass pasture grass, the percentage of grass weeds), the percentage of legumes (or the percentage of leguminous grass, the percentage of leguminous weeds), the percentage of weeds (or the percentage of grass weeds, the percentage of leguminous weeds, and other weeds), and the percentage of bare ground.

8. An information processing method for generating a training dataset, which uses the estimation result, which is the output result of a trained model generated by machine learning using high-resolution data of a predetermined target area photographed by a drone or UAV, as training data, and low-resolution data of a predetermined target area photographed by a satellite as input data, comprising: a high-resolution vegetation classification training dataset generation step of generating a high-resolution vegetation classification training dataset by associating high-resolution data of a target area photographed by a drone or UAV with training data indicating the vegetation classification of the target area; a first training processing step of performing machine learning processing to determine the vegetation classification from high-resolution data of a target area photographed by a drone or UAV using the generated high-resolution vegetation classification training dataset; and a high-resolution vegetation classification image generation step of estimating the vegetation classification of the target area and outputting a high-resolution vegetation classification image using a first trained model generated by the first training processing unit. An information processing method characterized by comprising: a vegetation ratio map generation step, which generates a vegetation ratio map that shows the proportion of the ground surface covered by a predetermined plant or the proportion of bare ground, based on the high-resolution vegetation classification image generated, wherein the vegetation ratio map is obtained by converting the low-resolution data captured by the satellite to a predetermined low resolution and dividing it into a grid (grid) section.

9. An information processing method for generating training data according to claim 8, comprising a step of generating a low-resolution vegetation ratio training dataset, wherein a low-resolution vegetation ratio training dataset is generated by associating a vegetation ratio map of the target area generated by the vegetation ratio map generation step with low-resolution data of the target area photographed by the satellite, as training data.

10. The information processing method according to claim 9, further comprising a second learning processing step of performing machine learning processing to determine the vegetation ratio from low-resolution data captured by satellite, using a low-resolution vegetation ratio learning dataset obtained by associating the vegetation ratio map of the target area generated in the vegetation ratio map generation step with low-resolution data of the target area captured by satellite as training data.

11. An information processing method for vegetation discrimination according to claim 10, comprising the steps of generating a trained model by performing machine learning processing in the second learning processing step using a low-resolution vegetation ratio learning dataset obtained by associating low-resolution data of the target area captured by the satellite with a vegetation ratio map of the target area generated by the vegetation ratio map generation unit as training data; and a classification step of inputting the low-resolution data captured by the satellite into the trained model and determining the vegetation of the target area based on the output from the trained model.

12. The information processing method according to claims 8 to 11, characterized in that, in the vegetation ratio map which is training data generated by the vegetation ratio map generation unit, the label value indicating the correct answer is represented by a vegetation ratio that includes one or more of the following: the ratio of plants covering the ground surface, the grass ratio (or grass ratio, legume ratio), the grass ratio (or grass ratio, grass weed ratio), the legume ratio (or legume ratio, legume weed ratio), the weed ratio (or grass weed ratio, legume weed ratio, other weed ratio), and the bare ground ratio.

13. A method for generating a trained model for a computer to determine the vegetation of a target area based on low-resolution data captured by a satellite, the information processing method according to claim 10, wherein in the second training step, machine learning processing is performed using a low-resolution vegetation ratio training dataset obtained by associating the vegetation ratio map of the target area generated in the vegetation ratio map generation step with the low-resolution data of the target area captured by a satellite as training data, thereby generating a second trained model.

14. A method for generating a second trained model according to claim 13, wherein the vegetation ratio map, which is training data generated by the vegetation ratio map generation unit, is characterized in that the label value indicating the correct answer is represented by a vegetation ratio that includes one or more of the following: the ratio of plants covering the ground surface, the percentage of pasture grass (or percentage of grassy pasture grass, percentage of leguminous pasture grass), the percentage of grass (or percentage of grassy pasture grass, percentage of grassy weeds), the percentage of leguminous plants (or percentage of leguminous pasture grass, percentage of leguminous weeds), the percentage of weeds (or percentage of grassy weeds, percentage of leguminous weeds, percentage of other weeds), and the percentage of bare ground.

15. A program for causing a computer to execute the information processing method described in any one of claims 8 to 11.

16. A program for causing a computer to execute the information processing method described in claim 12.

17. An information processing system for generating a training dataset, which uses an estimation result, which is the output result of a machine learning model that performs machine learning using high-resolution data of a predetermined target area, as training data, and low-resolution data of a predetermined target area as input data, comprising: a high-resolution attribute classification training data generation unit that generates a training dataset in which training data indicating the attribute classification of a target area is associated with high-resolution data of a predetermined target area; a first training processing unit that performs machine learning processing to determine the attribute classification of a target area from high-resolution data of a predetermined target area using the generated high-resolution attribute classification training dataset in which training data indicating the attribute classification of a target area is associated with high-resolution data of a predetermined target area; a high-resolution attribute classification image generation unit that estimates the attribute classification of the target area and outputs a high-resolution attribute classification image using a first trained model generated by the first training processing unit; and an attribute ratio map generation unit that generates an attribute ratio map, which is a map showing the ratio of a predetermined attribute of the target area, converted to a predetermined low resolution according to the resolution of the low-resolution data of the predetermined target area and divided into a grid-like section, based on the generated high-resolution attribute classification image.

18. An information processing system according to claim 17, further comprising: a second learning processing unit that performs machine learning processing to determine the ratio of a predetermined attribute of a target area from low-resolution data of a predetermined target area, using a low-resolution attribute ratio learning dataset obtained by associating the attribute ratio map of the target area generated by the attribute ratio map generation unit as training data with low-resolution data of the target area; a second learning processing unit that generates a second trained model by performing machine learning processing using a low-resolution attribute ratio learning dataset obtained by associating the attribute ratio map of the target area generated by the attribute ratio map generation unit as training data with low-resolution data of the target area; and a classifier that inputs the low-resolution data of the target area into the second trained model and determines the ratio of a predetermined attribute of the target area based on the output from the second trained model.

Images