Information processing system and information processing method

By integrating RTK positioning and deep learning to assign absolute coordinates to SAR images, the method addresses the inefficiencies of manual alignment and optical satellite limitations, facilitating rapid, high-precision three-dimensional modeling for disaster response and enhancing remote sensing data accuracy.

WO2026239687A1PCT designated stage Publication Date: 2026-11-19SOFTBANK CORPORATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/002243
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-13
Filing Date
2026-01-23
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Current methods for generating high-precision three-dimensional models from Synthetic Aperture Radar (SAR) images are time-consuming and costly due to the need for manual alignment of images lacking absolute coordinates, and optical satellite images are inadequate for real-time, all-weather observations.

Method used

A method that combines SAR images with Real-Time Kinematic (RTK) positioning technology and deep learning to generate pseudo-optical images with absolute coordinates, enabling efficient and accurate feature point matching and three-dimensional model creation.

Benefits of technology

Enables rapid, high-precision three-dimensional modeling over wide areas, supporting timely disaster response and improving remote sensing data accuracy, suitable for applications in telecommunications and achieving Sustainable Development Goal 9.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026002243_19112026_PF_FP_ABST
    Figure JP2026002243_19112026_PF_FP_ABST
Patent Text Reader

Abstract

An information processing system (Sy) comprises a first generation unit (133), an acquisition unit (131), an extraction unit (134), a matching unit (135), and an imparting unit (136). The first generation unit (133) generates an RGB image corresponding to a prescribed synthetic aperture radar image on the basis of the prescribed synthetic aperture radar image and a prescribed machine learning model. The acquisition unit (131) acquires a prescribed optical image which corresponds to the prescribed synthetic aperture radar image and to which first position information has been imparted. The extraction unit (134) extracts feature points in a first area indicated by the optical image and a second area indicated by the RGB image. The matching unit (135) matches the feature points between the first area and the second area. The imparting unit (136) imparts second position information corresponding to the first position information to the RGB image on the basis of the matching result.
Need to check novelty before this filing date? Find Prior Art

Description

Information Processing System and Information Processing Method

[0001] The present invention relates to an information processing system and an information processing method.

[0002] In recent years, observations utilizing three-dimensional data have been carried out.

[0003] For example, Patent Document 1 proposes a method for accurately detecting an aerial marker from a captured image obtained by photographing the aerial marker, for the purpose of measuring buildings, earth volume, and the like.

[0004] Japanese Unexamined Patent Application Publication No. 2022-133328

[0005] The information processing system according to the present application includes: a first generation unit configured to generate an RGB image corresponding to a predetermined synthetic aperture radar image based on the predetermined synthetic aperture radar image and a predetermined machine learning model; an acquisition unit configured to acquire a predetermined optical image corresponding to the predetermined synthetic aperture radar image, the optical image being provided with first position information; an extraction unit configured to extract feature points in a first area indicated by the optical image and a second area indicated by the RGB image; a matching unit configured to match the feature points between the first area and the second area; and an assignment unit configured to assign, to the RGB image, second position information corresponding to the first position information based on a result of the matching.

[0006] Figure 1 is a diagram showing an overview of ground surface observation using SAR interferometry. Figure 2 is a diagram showing the overall picture of absolute coordinate assignment processing. Figure 3 is a diagram showing the overall picture of 3D conversion processing. Figure 4 is a diagram showing an example of the configuration of an information processing device according to the embodiment. Figure 5 is a diagram showing an example of a learning method according to the embodiment. Figure 6 is a diagram showing a specific example of feature point matching according to the embodiment. Figure 7 is a flowchart showing the procedure for preprocessing PR1. Figure 8 is a flowchart showing the procedure for preprocessing PR2. Figure 9 is a diagram showing a specific example of location information assignment. Figure 10 is a flowchart showing the procedure for colorization processing. Figure 11 is a flowchart showing the procedure for feature point matching. Figure 12 is a flowchart (1) showing the procedure for 3D conversion processing. Figure 13 is a flowchart (2) showing the procedure for 3D conversion processing. Figure 14 is a flowchart showing the procedure for calculating the volume of soil and sediment. Figure 15 is a flowchart showing the procedure for control processing for soil and sediment removal. Figure 16 is a diagram showing an overall picture of a modified example. Figure 17 is a flowchart showing the procedure for feature point matching related to the modified example. Figure 18 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device 100.

[0007] Embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0008] The one or more embodiments (including examples, modifications, and applications) described below can each be implemented independently. On the other hand, at least some of the embodiments described below may be implemented in appropriate combination with at least some of the other embodiments. These embodiments may contain novel features that differ from each other. Therefore, these embodiments may contribute to solving different objectives or problems and may produce different effects.

[0009] In the following embodiments, the term "optical image" includes the concept of optical satellite images taken by an optical satellite, or aerial images taken from above by an aircraft such as a drone. Furthermore, although the events targeted by the proposed technology of the present invention (hereinafter sometimes simply referred to as "the proposed technology") are described as natural disasters caused by critical natural phenomena, the proposed technology of the present invention can be applied to various events, not limited to natural disasters.

[0010] Natural disasters include storms, heavy rain, heavy snow, floods, storm surges, earthquakes, tsunamis, and volcanic eruptions, but the following embodiment will describe the proposed technology focusing on landslides.

[0011] <Embodiment> [1. Introduction] (About SAR) Observation using Synthetic Aperture Radar (SAR) is well known. SAR is not a passive sensor that utilizes electromagnetic wave energy radiated from the sun, but an active sensor that irradiates microwaves from the radar antenna itself toward the object to be observed, and receives the electromagnetic wave energy scattered by the object with the antenna, so it can observe regardless of day or night. In addition, SAR irradiates microwaves with wavelengths longer than the size of particles in the atmosphere such as clouds and rain, so it is less affected by cloud cover, etc. Thus, SAR has the characteristic of being all-weather, and has the advantage of being able to observe regardless of the time of day or weather conditions.

[0012] Furthermore, because SAR can capture objects with a spatial resolution unattainable by other observation methods, it is also used to observe sediment-related disasters such as landslides and ground surface deformation caused by subsidence. For example, a satellite-mounted SAR irradiates microwaves perpendicular to the satellite's direction of travel and obliquely downwards, observes the backscattered waves from the ground surface, and analyzes the observed data to obtain a SAR image with high spatial resolution.

[0013] SAR images store information on scattering intensity and phase for each pixel. The scattering intensity includes information about the slope, roughness, and dielectric constant of the Earth's surface, while the phase includes information about the distance from the satellite to the pixel and the scattering pattern. Thus, SAR images are intensity images that include phase information, but it is difficult to observe changes in the Earth's surface using only one image.

[0014] Therefore, a technique called SAR interferometry (interferometric SAR) is known. SAR interferometry is a method that observes the shape and changes of the Earth's surface at an observation site by observing the same location multiple times using a satellite-mounted SAR. Here, Figure 1 shows an overview of Earth's surface observation using SAR interferometry. Figure 1 shows a scenario in which two SAR observations are performed on the same location on the Earth's surface using the SAR satellite 10, an artificial satellite equipped with SAR.

[0015] In this way, by performing two SAR observations to prepare two SAR images (phase images) and then interfering them and taking the difference, it becomes possible to utilize information from small distance differences Δd through interferometry (also called SAR analysis or interferometry), making it possible to capture surface deformation. Furthermore, the results of interferometry are generally represented as an interferogram, which shows deformation as rainbow-colored stripes. As shown in the example in Figure 1, the interferogram is created by interfering the SAR image from the first observation with the SAR image from the second observation, and shows an interferogram with interference fringes such as topographic fringes and orbital fringes. Thus, with SAR interferometry, for example, by performing interferometry using two observation data acquired before and after an earthquake, it becomes possible to visualize crustal deformation associated with earthquakes.

[0016] SAR images have several processing levels, but the data immediately after imaging, which serves as the basis for subsequent processing, is also called SLC (Single Look Complex) data. SLC data is the SAR image with the highest resolution and includes not only amplitude information that can be recognized as an image, but also phase information, etc.

[0017] For example, in interferometry, a differential interferometry image is generated using SLC data obtained from the first observation and SLC data obtained from the second observation, following the procedure below. Specifically, the procedure involves aligning the SLC data obtained from the first observation with the SLC data obtained from the second observation (Step 1), performing interferometry to generate an interference image from the two SLC data (Step 2), and generating a differential interferometry image by removing interference fringes from the interference image (Step 3).

[0018] In step 1, the alignment process involves measuring the corresponding pixel coordinate difference at the sub-pixel level based on the SLC data obtained from the first and second observations. This calculates a conversion coefficient that determines the pixel position in the SLC data obtained from the second observation that corresponds to the pixel in the SLC data obtained from the first observation. Based on this conversion coefficient, the two SLC data sets are aligned. However, this type of alignment presents a problem: it requires time-consuming tasks such as manually overlaying landmarks. The background and challenges leading to the proposed technology are explained in detail below.

[0019] (Problem 1) Rapid response is required when a disaster occurs, and by comparing three-dimensional models (three-dimensional point cloud data) acquired by laser scanners, it is possible to distinguish between damaged and undamaged areas, for example, and its usefulness in countermeasures and rescue efforts by local governments has been found. However, there are limitations to the means of generating high-precision three-dimensional models of the entire country (for example, all of Japan), and while there is a need to rapidly generate high-precision three-dimensional models over a wide area, current methods are time-consuming and costly.

[0020] Therefore, the inventors of this invention focused on the possibility that a high-precision 3D model could be generated using SAR images as source images if a SAR capable of capturing high-resolution images in all weather conditions, regardless of time of day or weather conditions, were used. Furthermore, SAR missions capable of observation at higher intervals are now being realized, and multiple SAR satellites 10 are being operated simultaneously. As a result, it is possible to observe the same location multiple times a day using multiple SAR satellites 10, and SAR is considered useful in terms of accelerating disaster response.

[0021] On the other hand, when generating a three-dimensional model from SAR images, alignment of the SAR images is performed, but as mentioned above, this alignment requires manual work, which is time-consuming and costly. This is because SAR images do not have absolute coordinates, making it difficult to determine their position. For example, while relative coordinate changes can be monitored from phase changes on different time axes in SAR images, the SAR images themselves do not have absolute coordinates. For example, the management company of SAR satellite 10 performs the laborious task of overlaying SAR images to represent coordinates based on the positions of landmarks and other elements contained in the SAR images.

[0022] In this regard, the inventors of the present invention have found that absolute coordinates can be assigned to SAR images by combining RTK (Real Time Kinematic) positioning technology, which can determine positional information with high accuracy of an error of a few centimeters, with SAR observation technology.

[0023] (Problem 2) From here, we will explain the background and challenges that led to the proposed technology from a different perspective than the above. For example, one possible method for assigning absolute coordinates to SAR images is to extract feature points of the shooting area from both an optical image, which has absolute coordinates based on the position coordinates of a reference station used for RTK positioning, and a SAR image, which does not have absolute coordinates, and then assign absolute coordinates to the SAR image by feature point matching using the extracted feature points.

[0024] However, accurate feature point matching may not be possible between optical images and SAR images. Specifically, the principle of optical imaging involves using light-based remote sensing technology to measure scattered light from sunlight and analyze the properties of an object measured at a distance. For this reason, optical images are generated as full-color RGB images.

[0025] On the other hand, the principle behind SAR imaging involves analyzing the properties of an object measured at a distance by observing the reflected waves when radio waves are shone onto the object. For this reason, SAR images are generated as grayscale black and white images.

[0026] Thus, because optical images and SAR images use different imaging principles, proper feature point matching may not always be possible. In this case, it is conceivable to generate a three-dimensional model using only optical images, without using SAR images. However, optical satellites have various drawbacks. As an example of optical satellites, we will explain these drawbacks using optical satellite images acquired by optical satellites.

[0027] For example, optical satellites have significantly fewer launches compared to SAR satellites, and the rate at which optical satellite images of the same location can be acquired is about one image per 14 days, which is slower than the rate at which SAR images of the same location can be acquired (about one image per 10 minutes). Therefore, the use of optical satellite images lacks real-time capabilities and is impractical in cases where a rapid response is required, such as during a disaster. In addition, optical satellite images are susceptible to the effects of time of day and weather conditions, as they cannot be observed in cloudy areas or at night.

[0028] For this reason, the inventors of the present invention have focused on the use of SAR images. As described above, the SAR satellite 10 can observe the same location at a rate of about one image per 10 minutes. Therefore, the use of SAR images has the advantage of being excellent in real time and being able to be observed regardless of the time of day or weather conditions, thus covering the shortcomings of using optical satellite images. However, SAR images have the problem that they are not suitable for feature point matching to assign absolute coordinates.

[0029] (Overview of the proposed technology) Based on the above, the proposed technology first generates a pseudo-optical image, which is a color image (i.e., an RGB image) that closely resembles the color tones of an optical image, by colorizing a grayscale SAR image. In this state, the proposed technology extracts feature points of the shooting area from both the optical image, which has absolute coordinates, and the pseudo-optical image, which does not have absolute coordinates, and then assigns absolute coordinates to the pseudo-optical image by feature point matching using the extracted feature points. As a result, the proposed technology enables positional identification of the pseudo-optical image, and a three-dimensional model is generated from the pseudo-optical image to which absolute coordinates have been assigned.

[0030] This proposed technology will enable more efficient and high-precision observation using 3D data.

[0031] Furthermore, the proposed technology is a novel technology that improves the accuracy of remote sensing data, including SAR images, by correcting the absolute coordinates of optical images. It overcomes the problem that conventional SAR images do not have absolute coordinates, and enables highly accurate three-dimensional analysis and the creation of GIS data, thus having extremely broad industrial applications. Therefore, the proposed technology can serve as an innovative technological foundation in, for example, the telecommunications industry, and thus contribute to achieving Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0032] [2. Overview of the Proposed Technology] From here, the overall picture of the proposed technology of the present invention will be explained using Figures 2 and 3. Figure 2 explains the overall picture of the absolute coordinate assignment process in which absolute coordinates are assigned to SAR images. Figure 3 explains the overall picture of the 3D conversion process in which a three-dimensional model is generated from the SAR image to which absolute coordinates have been assigned.

[0033] (Absolute Coordinate Assignment Processing) Figure 2 is a diagram showing the overall structure of the absolute coordinate assignment processing. Figure 2 shows an information processing system Sy in which the information processing related to the proposed technology is realized. The information processing system Sy may include a SAR observation system 1 that performs SAR observations using a SAR mounted on the SAR satellite 10, and an RTK positioning system 2. The mechanism of SAR observation in the SAR observation system 1 is as described above, so the explanation is omitted.

[0034] The RTK positioning system 2 may include a receiving device 20, a reference station 30, a GNSS (Global Navigation Satellite System) satellite 40, and a positioning core system 50.

[0035] The receiving device 20 may be a portable information processing terminal and can be installed, for example, at any location where a user wishes to know their location information. The receiving device 20 may receive GNSS signals from the GNSS satellite 40. That is, the receiving device 20 may be equipped with, for example, a GNSS module (positioning module) and an antenna corresponding to a GNSS receiver in RTK. The receiving device 20 may also be equipped with a communication module for communicating with the positioning core system 50.

[0036] The reference station 30 may function as a reference station in RTK calculations. That is, the reference station 30 may have known coordinates that indicate its own position. Furthermore, if there are multiple reference stations 30, each of the multiple reference stations 30 may have known coordinates that are defined for it. The reference station 30 may receive GNSS signals from the GNSS satellite 40.

[0037] In RTK positioning, first, the receiving device 20 and multiple reference stations 30 whose precise position information (absolute coordinates) is known receive GNSS signals from the GNSS satellite 40 and each acquires its own position information based on the received GNSS signals. Next, the receiving device 20, which is trying to determine its position, transmits its approximate position information (hereinafter referred to as approximate position information) obtained based on the GNSS signals to the positioning core system 50. Then, based on the transmitted approximate position information, the positioning core system 50 acquires the position information of the multiple reference stations 30 located near the receiving device 20 in real time, obtained based on the GNSS signals of those reference stations 30.

[0038] Next, the positioning core system 50 generates correction information by calculating the difference between known, precise position information of the reference station 30 (for example, pre-stored in the positioning core system 50) and position information obtained from the reference station 30 based on the GNSS signal. This correction information is used to correct the approximate position information of the receiving device 20 obtained based on the GNSS signal in real time. In other words, in RTK positioning, the approximate position information of the receiving device 20 obtained based on the GNSS signal is corrected using correction information generated from the position information of the reference station 30 (position information obtained based on the GNSS signal, precise position information) installed in a receiving environment that is estimated to be close to the GNSS signal receiving environment of the receiving device 20 whose position is to be determined.

[0039] With this type of RTK positioning, by performing the corrections described above, it is possible to correct errors caused by ionospheric disturbances, multipath, and clock drift in the GNSS signal, thereby reducing the error included in the position information to about a few centimeters. As a result, RTK positioning makes it easy to acquire position information of the receiving device 20 with minimal error.

[0040] In the RTK positioning system used in this embodiment, it is preferable that the multiple reference stations 30 are set up so that the distance between them is within 10 kilometers.

[0041] Further, in RTK positioning, the positioning core system 50 or the receiving device 20 uses the correction information generated by the positioning core system 50 to correct the approximate position information obtained based on the GNSS signal, thereby acquiring the position information of the receiving device 20 with less error. More specifically, RTK positioning can be mainly divided into two schemes: a device RTK positioning scheme in which the above correction is performed on the receiving device 20 side, and a server RTK positioning scheme in which the above correction is performed in the positioning core system 50.

[0042] In the absolute coordinate assignment processing implemented by the information processing system Sy, known accurate position information is used for each of the plurality of reference stations 30.

[0043] Here, as shown in FIG. 2, the reference station image BG including the reference station 30 is an optical satellite image including the reference station 30 captured by the optical satellite 60, and belongs to a full-color optical image. In addition, the reference station image BG has the position information of the reference station 30. As described above, the position information of the reference station 30 is accurate position information predetermined for the reference station 30, and is absolute position information (absolute coordinates) for specifying the position of the first area AR (hereinafter referred to as "area AR1"), which is the imaging area AR indicated by the reference station image BG. Note that the number of reference stations 30 included in the reference station image BG is not limited.

[0044] In such a state, in the absolute coordinate assignment processing, first position information based on the accurate position information of the reference station 30 is assigned to the reference station image BG. Specifically, in the absolute coordinate assignment processing, position information that is based on the accurate position information of the reference station 30 and that relates to objects around the reference station 30 included in the area AR1 is calculated as the first position information. Then, the first position information is assigned to the image portion of the object in the reference station image BG. The first position information can be said to be absolute position information (absolute coordinates) based on the accurate position information of the reference station 30.

[0045] In addition, the SAR satellite 10 performs imaging while orbiting along a specific orbit. Therefore, sequential SAR observation corresponding to the orbit can obtain a nationwide (for example, the entire territory of Japan) SAR image SG, similar to the reference station image BG. The SAR image SG itself does not have absolute coordinates, and absolute coordinates are assigned thereto through absolute coordinate assignment processing.

[0046] In the absolute coordinate assignment processing, feature point matching with the reference station image BG is performed to assign absolute coordinates to the SAR image SG. However, as described above, the reference station image BG which is an optical image and the SAR image SG which is a grayscale image have different imaging principles, so there are cases where appropriate feature point matching cannot be performed.

[0047] Accordingly, in the absolute coordinate assignment processing, the SAR image SG is colorized to generate a pseudo-optical image CL_SG which is a full-color image (i.e., an RGB image) approximate to the tint of an optical image. Specifically, in the absolute coordinate assignment processing, a pseudo-optical image CL_SG obtained by colorizing the SAR image SG is generated based on the model M1, which is an AI model trained to pseudo-reproduce an optical image corresponding to the input SAR image SG, and the SAR image SG that is the target for absolute coordinate assignment. Hereinafter, the SAR image SG that is the target for absolute coordinate assignment (the SAR image SG that is not colorized and also has no absolute coordinates) is referred to as "original SAR image SG".

[0048] FIG. 2 shows an example in which a pseudo-optical image CL_SG having the imaging area AR as the area AR2 is generated corresponding to a second area AR (hereinafter referred to as "area AR2") which is the imaging area AR indicated by the original SAR image SG.

[0049] At this point, a reference station image BG with absolute coordinates and a pseudo-optical image CL_SG without absolute coordinates are obtained. Then, in the absolute coordinate assignment process, feature points F in area AR1 indicated by the reference station image BG and area AR2 indicated by the pseudo-optical image CL_SG without absolute coordinates are extracted. For example, in the absolute coordinate assignment process, the first feature point F (hereinafter referred to as "feature point F1"), which is a feature point F in area AR1, is extracted, and the second feature point F (hereinafter referred to as "feature point F2"), which is a feature point F in area AR2, is extracted.

[0050] For feature point extraction, a predetermined deep learning-based AI model, Model M2, may be used. Model M2 may be, for example, SIFT (Scale-Invariant Feature Transform), AKAZE (Accelerated-KAZE), ORB (Oriented FAST and Rotated BRIEF), etc.

[0051] Next, in the absolute coordinate assignment process, a matching process is performed to match feature points F1 and F2 with common feature points F.

[0052] Then, in the absolute coordinate assignment process, based on the results of the matching process, second position information corresponding to the first position information is assigned to the pseudo-optical image CL_SG which does not have absolute coordinates. For example, in the absolute coordinate assignment process, information transfer may be performed to transfer the second position information to all feature points F2 included in a pair, based on the feature point pair matched between feature point F1 and feature point F2, and the feature point F2 matched with feature point F1 which points to the object for which the first position information was calculated. In addition, in the absolute coordinate assignment process, the transfer of second position information on a pixel-by-pixel basis may also be performed using the second position information assigned to all feature points F2. The second position information, like the first position information, can be said to be absolute position information (absolute coordinates) based on the accurate position information of the reference station 30.

[0053] Figure 2 shows an example in which a pseudo-optical image CL_SG' is obtained as a pseudo-optical image CL_SG having absolute coordinates through an absolute coordinate assignment process.

[0054] (3D Processing) Figure 3 shows an overview of the 3D processing. The 3D processing may also be performed in the information processing system Sy. As shown in Figure 3, the 3D processing may be performed using the original SAR image SG and a pseudo-optical image CL_SG' in which the original SAR image SG has been colorized and absolute coordinates have been assigned. In addition, the 3D processing described here requires processing to determine the condition of the topography of the ground surface (specifically, processing to detect the area of ​​object OB included in the image and to determine the height of the object in the detected area), and the obtained information is added to the pseudo-optical image CL_SG' obtained in the absolute coordinate assignment processing.

[0055] Let's explain this in more detail using the example in Figure 3. In the 3D conversion process, as shown in Figure 3, Model M3, an AI model trained to detect the region of a specific object OB (for example, a structure such as a house or building) from a SAR image on a pixel-by-pixel basis, and Model M4, an AI model trained to estimate the height of the region detected from the SAR image, may be used.

[0056] Model M3 can be a so-called semantic segmentation model and can be called a "building extraction AI." Model M4 can be called a "height inference AI." Both Model M3 and Model M4 are machine learning models newly trained and generated to realize the proposed technology, and may be, for example, a CNN (Convolutional Neural Network).

[0057] In the example shown in Figure 3, during the 3D conversion process, the region of object OB in area AR2 indicated by the original SAR image SG is detected based on the output result output by model M3 for the input original SAR image SG, and object region information OB_DA2, which is information about the detected region, is obtained.

[0058] In the example shown in Figure 3, during the 3D conversion process, the height of object OB in area AR2 indicated by the original SAR image SG is estimated based on the output result output by model M4 for the input original SAR image SG, and object height information OB_DA3, which is information about the estimated height, is obtained.

[0059] Furthermore, in the 3D conversion process, object information OB_DA, which includes object region information OB_DA2 and object height information OB_DA3, is generated. Then, by associating the object information OB with the original SAR image SG, a SAR image OB_SG having object information OB_DA is synthesized. As shown in Figure 3, the SAR image OB_SG has object information OB_DA but does not have absolute coordinates.

[0060] At this point, a SAR image OB_SG containing object information OB_DA and a pseudo-optical image CL_SG' containing absolute coordinates are obtained. Then, in the 3D conversion process, feature points F in area AR2 indicated by the SAR image OB_SG and in area AR2 indicated by the pseudo-optical image CL_SG' are extracted. For example, in the 3D conversion process, the third feature point (hereinafter referred to as "feature point F3") which is a feature point on the SAR image OB_SG side in area AR2, and the fourth feature point (hereinafter referred to as "feature point F4") which is a feature point on the pseudo-optical image CL_SG' side in area AR2 are extracted. Model M2 may be used as the algorithm for feature point extraction.

[0061] Next, in the 3D modeling process, a matching process is performed to match feature points F3 and F4 with common feature points F.

[0062] Then, in the 3D conversion process, object information OB_DA is assigned to the pseudo-optical image CL_SG' which has absolute coordinates, based on the results of the matching process. For example, in the 3D conversion process, information transfer may be performed to transfer the object information OB_DA to all feature points F4 included in the pair, based on the feature point pair matched between feature point F3 and feature point F4, and the object information OB_DA associated with the image portion of the paired feature point F3.

[0063] Figure 3 shows an example in which a pseudo-optical image CL_SG'' is obtained as a pseudo-optical image CL_SG that further has object information OB_DA in addition to absolute coordinates through 3D processing.

[0064] In this state, the 3D conversion process generates a three-dimensional model MD in which the object OB, which is represented in a plane in the pseudo-optical image CL_SG'', is represented three-dimensionally, based on the object information OB_DA attached to the pseudo-optical image CL_SG''.

[0065] Furthermore, in the 3D conversion process, the final three-dimensional model MD' may be generated by applying processing to the three-dimensional model MD to bring it closer to the actual colors, according to the RGB elements contained in the pseudo-optical image CL_SG''.

[0066] The information processing according to the embodiment described in Figures 2 and 3 may be performed by the information processing device 100, which will be described later. Therefore, the information processing device 100 makes it possible to improve the efficiency of observations using three-dimensional data. For example, since the information processing device 100 is introduced into an information processing system Sy that includes a SAR observation system 1 and an RTK positioning system 2, it is possible to perform information processing that combines RTK positioning technology and SAR observation technology, making it possible to quickly generate high-precision three-dimensional models over a wide area without spending time and money. As a result, the information processing device 100 can support a rapid response in the event of a disaster. For example, if a landslide occurs due to an earthquake, the information processing device 100 can estimate the type and number of equipment necessary for removing the sediment based on the volume of the sediment and notify the local government of the estimation results.

[0067] [3. Configuration of Information Processing Device] An information processing device 100 according to the embodiment will be described using Figure 4. Figure 4 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Figure 4, the information processing device 100 may have a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may be a server device implemented in the cloud.

[0068] (Communication Unit 110) The communication unit 110 is implemented by, for example, a NIC (Network Interface Card). For example, the communication unit 110 transmits and receives information between the SAR observation system 1 and the RTK positioning system 2. The communication unit 110 may also transmit and receive information between the user U's terminal device DV (not shown). The user U referred to here may be, for example, an employee of a local government in the area where the disaster occurred.

[0069] (Storage Unit 120) The storage unit 120 is implemented by, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 120 may store, for example, data and programs related to the information processing according to the embodiment. The storage unit 120 may also store data necessary for the information processing according to the embodiment, or data obtained from the information processing according to the embodiment.

[0070] (Control Unit 130) The control unit 130 is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc., which executes various programs (for example, the information processing program according to the embodiment) stored in the storage device inside the information processing device 100 using RAM as the working area. The control unit 130 is also implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0071] As shown in Figure 3, the control unit 130 includes an acquisition unit 131, a learning unit 132, a first generation unit 133, an extraction unit 134, a matching unit 135, an assignment unit 136, a second generation unit 137, an estimation unit 138, and a notification unit 139, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 4, and other configurations are also possible as long as they perform the information processing described later. Also, the connection relationships of the various processing units in the control unit 130 are not limited to the connection relationships shown in Figure 4, and other connection relationships are also possible.

[0072] (Acquisition Unit 131) The acquisition unit 131 acquires information necessary for the information processing according to the embodiment. For example, the acquisition unit 131 acquires optical images and SAR images.

[0073] (Learning Unit 132) The learning unit 132 trains a machine learning model for generating image data. For example, the learning unit 132 may use a combination of a sample SAR image and a sample optical image as training data to train model M on the relationship between the two images, thereby generating an AI model (model M1) that simulates an optical image corresponding to an input SAR image. The learning unit 132 may use a Diffusion Model as model M. The learning unit 132 may also train model M3, which is a building extraction AI, and model M4, which is a height inference AI.

[0074] (First Generation Unit 133) The first generation unit 133 generates a full-color RGB image from a grayscale SAR image. Specifically, the first generation unit 133 uses a combination of sample SAR images and sample optical images as training data, with the combination of sample SAR images and sample optical images being used as training data to input an original SAR image (an example of a predetermined SAR image) into the AI ​​model (Model M1), which has been trained to simulate an optical image corresponding to the input SAR image. The first generation unit 133 generates a pseudo-optical image (RGB image) in which the optical image corresponding to the original SAR image is simulated.

[0075] (Extraction Unit 134) The extraction unit 134 extracts feature points from the image data. For example, the extraction unit 134 extracts feature points in the first area shown by the optical image and the shooting area shown by the pseudo-optical image. The extraction unit 134 also extracts feature points from each image in the second area that is common to both the SAR image having object information and the pseudo-optical image generated from the original SAR image.

[0076] The extraction unit 134 may use SIFT, AKAZE, ORB, etc., as the algorithm (model M2) for extracting feature points. For example, SIFT is one of the feature point detection and feature description algorithms. It detects feature points from the difference of a smoothed image obtained by convolving with DoG (Differences of Gaussian), which approximates LOG (Laplacian of Gaussian), and describes them as features using a 128-dimensional gradient vector obtained from the surrounding pixel information. With SIFT, features can be described robustly to image rotation, scale changes, lighting changes, etc., for the detected feature points, and can therefore be used for image matching such as image mosaics, as well as object recognition and detection.

[0077] (Matching Unit 135) The matching unit 135 performs feature point matching to match feature points between different image data. For example, the matching unit 135 performs feature point matching between different image data based on feature points and feature quantities. For example, the matching unit 135 calculates a norm as the feature quantity of each feature point and matches the feature points that have the minimum distance from each other between the images as common feature points.

[0078] For example, the matching unit 135 performs feature point matching between an optical image and a pseudo-optical image. The matching unit 135 also performs feature point matching between multiple pseudo-optical images that have different assigned information. For example, the matching unit 135 performs feature point matching between an SAR image that has object information relative to the original SAR image and a pseudo-optical image generated from the original SAR image.

[0079] (Information Assignment Unit 136) The information assignment unit 136 assigns information to the image data. For example, the information assignment unit 136 assigns first position information, which is absolute position information based on the precise position information of the reference station 30 and is assigned to the reference station image, to the reference station image BS (optical satellite image).

[0080] Furthermore, the assignment unit 136 assigns information to the image data based on the matching results from the matching unit 135. For example, the assignment unit 136 assigns second position information corresponding to the first position information to the pseudo-optical image (RGB image) as absolute position information based on the precise position information of the reference station 30. The assignment unit 136 also assigns object information to the SAR image to the pseudo-optical image generated from the original SAR image.

[0081] (Second generation unit 137) The second generation unit 137 generates a three-dimensional model that virtually represents the shooting area shown by the pseudo-optical image and has the second position information, based on the original SAR image and the pseudo-optical image to which the second position information is attached.

[0082] As described above, the assignment unit 136 assigns object information to the pseudo-optical image based on the result of matching feature points between the SAR image containing object information and the pseudo-optical image generated from the original SAR image. Then, the second generation unit 137 generates a three-dimensional model having second position information based on the pseudo-optical image to which the object information has been assigned.

[0083] (Estimation Unit 138) The estimation unit 138 performs various estimations. For example, the estimation unit 138 detects the soil and sediment based on a three-dimensional model generated at different timings along a time series, and estimates the volume of the detected soil and sediment. The estimation unit 138 may also estimate the type and number of equipment necessary for soil and sediment removal based on the estimated volume of soil and sediment.

[0084] For example, the estimation unit 138 may estimate the type and number of equipment required for soil removal in a rule-based manner. Specifically, if the type and number of equipment required for soil removal are defined as estimation rules for each volume of soil, the estimation unit 138 may estimate the type and number of equipment required for soil removal by comparing the current volume of soil with the estimation rules.

[0085] As another example, the estimation unit 138 may estimate the type and number of equipment required for soil removal based on a machine learning model. In this case, the machine learning model may be trained to output the type and number of equipment required for soil removal when the volume of soil is input.

[0086] (Regarding the notification unit 139) The notification unit 139 provides various types of information to the user U. For example, if the notification unit 139 determines that an event has occurred, it may notify the user U of information regarding the event that occurred. For example, if the notification unit 139 determines that a landslide has occurred, it may notify the user U of alert information that includes at least information indicating the occurrence of a landslide and information about the area where the landslide occurred.

[0087] Furthermore, the notification unit 139 may notify the user U of information regarding the volume of sediment, the geographical area newly covered by sediment due to the sediment disaster (sediment disaster area), and the types and number of equipment necessary for sediment removal.

[0088] [4. Specific Examples of Learning Methods] Using Figure 5, a specific example of a learning method for generating an AI model (Model M1) that simulates an optical image corresponding to an input SAR image will be explained. Figure 5 is a diagram showing an example of a learning method according to the embodiment. Figure 5 shows a scene in which Model M1 is generated using a combination of a sample SAR image SM_SG and a sample optical image SM_TG as training data LD.

[0089] As shown in the example in Figure 5, the learning unit 132 may generate a dataset consisting of multiple training data LDs, where each pair of optical image SM_TG and SAR image SM_SG, whose imaging areas are similar (or common), is treated as a single training data LD.

[0090] The learning unit 132 may then input the dataset into a model M, which is a pre-trained diffusion model, to train it to estimate RGB elements on a pixel-by-pixel basis for grayscale SAR images.

[0091] For example, as shown in Figure 5, the learning unit 132 may generate a model M1 that outputs a pseudo-optical image CL_SG, which is a pseudo-optical image that simulates an optical image corresponding to the input SAR image SG, by having the model M learn the relationship between RGB and grayscale between the optical image SM_TG and the SAR image SM_SG.

[0092] [5. Specific Examples of Feature Point Matching] Figure 6 shows a specific example of feature point matching according to the embodiment. Before explaining feature point matching, Figure 6(a) will be used to explain wide-area observation by the SAR satellite 10.

[0093] The SAR satellite 10 orbits the Earth, and its orbit includes a path descending from north to south (southern orbit) and a path ascending from south to north (northern orbit). When performing feature point matching between the reference station image BG and the SAR image SG, it is preferable that the observation ranges of the reference station image BG and the SAR image SG are the same or partially overlap.

[0094] Figure 6(a) shows the SAR satellite 10 performing SAR observations of the ground L while moving along a single orbit TR.

[0095] In the example shown in Figure 6(a), the SAR satellite 10 observes areas R1, R2, R3, and R4 by performing wide-area observations from position P1 on the orbit TR. As a result, the information processing device 100 can obtain SAR images SG taken from each of the areas R1, R2, R3, and R4.

[0096] Furthermore, as shown in the example in Figure 6(a), the SAR satellite 10 observes areas R5, R6, R7, and R8 by performing wide-area observations from position P2 on the orbit TR. As a result, the information processing device 100 can obtain SAR images SG taken from each of the areas R5, R6, R7, and R8.

[0097] Furthermore, as shown in the example in Figure 6(a), the SAR satellite 10 observes areas R9, R10, R11, and R12 by performing wide-area observations from position P3 on the orbit TR. As a result, the information processing device 100 can obtain SAR images SG taken from areas R9, R10, R11, and R12, respectively.

[0098] Furthermore, since the SAR satellite 10 orbits the Earth along its orbit TR and observes the same observation range at predetermined intervals, the information processing device 100 can sequentially acquire SAR images SG of range R1 at a rate of, for example, 6 images per hour (144 images per 24 hours). The same applies to ranges R2 to R12.

[0099] Next, a specific example of feature point matching according to the embodiment will be explained using Figure 6(b). Figure 6(b) shows a scenario in which feature point matching is performed between a reference station image BG1 and a SAR image SG1, both of which share the same observation range R1. Thus, area AR1 shown in the reference station image BG1 corresponds to the area R1. Similarly, area AR2 shown in the SAR image SG1 also corresponds to the area R1.

[0100] Furthermore, according to the example in Figure 6(b), the reference station image BG1 includes a reference station 31 (an example of a reference station 30) installed on structure ST1 (an example of object OB). Similarly, the SAR image SG1 also includes a reference station 31 installed on structure ST1. In the example in Figure 6(b), first position information, based on the precise position information of reference station 31, is assigned to the location of structure ST1 within the reference station image BG1. In this example, the first position information is the absolute coordinates of structure ST1. Therefore, in the example in Figure 6(b), the reference station image BG1 has absolute coordinates. On the other hand, the SAR image SG1 does not have absolute coordinates in its original state as acquired. Specifically, the SAR image SG1 does not have absolute coordinates, which are the absolute position information of structure ST1. Also, the SAR image SG1 is grayscale in its original state.

[0101] Therefore, in the absolute coordinate assignment process for assigning absolute coordinates, prior to feature point matching, the original SAR image SG1 is colorized to generate a pseudo-optical image CL_SG, which is a full-color image (i.e., an RGB image) that closely resembles the color tones of the reference station image BG1, which is an optical satellite image. Specifically, the first generation unit 133 inputs the grayscale original SAR image SG1 to the model M1, and based on the output result of the model M1, generates a pseudo-optical image CL_SG1 in which the optical image corresponding to the original SAR image SG1 is pseudo-reproduced.

[0102] At this point, a reference station image BG1 with absolute coordinates and a pseudo-optical image CL_SG1 without absolute coordinates are obtained. Then, in the absolute coordinate assignment process, feature points F in area AR1 indicated by the reference station image BG1 and area AR2 indicated by the pseudo-optical image CL_SG1 without absolute coordinates are extracted. Specifically, the extraction unit 134 uses model M2 to extract feature point F1, which is the first feature point F in area AR1, and also extracts feature point F2, which is the second feature point F in area AR2.

[0103] In this state, the absolute coordinate assignment process performs a matching process to match feature point F1 and feature point F2 with a common feature point F. Specifically, the matching unit 135 calculates a norm as the feature quantity for feature point F1 and feature point F2, and matches the feature point F that minimizes the distance between the reference station image BG1 and the pseudo-optical image CL_SG1, i.e., between feature point F1 and feature point F2, as the common feature point F.

[0104] Then, in the absolute coordinate assignment process, based on the results of the matching process, second position information corresponding to the first position information is assigned to the pseudo-optical image CL_SG1 which does not have absolute coordinates. Specifically, the assignment unit 136 performs information transfer to transfer the second position information to all feature points F2 included in the pair, based on the feature point pair matched between feature point F1 and feature point F2, the feature point F2 that matches with feature point F which points to structure ST1, and the first position information assigned to the location of structure ST1 in the reference station image BG1. The assignment unit 136 may further assign second position information to the pseudo-optical image CL_SG1 on a pixel-by-pixel basis using the second position information assigned to all feature points F2.

[0105] [6. Processing Procedure] From here, a specific example of the operation of the information processing device 100 will be described. The information processing according to this embodiment may be divided into absolute coordinate assignment processing, 3D conversion processing, and processing as a service utilizing the three-dimensional model. In addition, in the absolute coordinate assignment processing, pre-processing for assigning absolute coordinates may be performed first. The specific operation procedures of the information processing device 100 in each process will be described below.

[0106] (6-1. Preprocessing (1)) Figure 7 is a flowchart showing the procedure for preprocessing PR1. Preprocessing PR1 may be a process that uses the precise location information of the base station 30 to transfer the location information to object OB.

[0107] As described above, each of the base stations 30 in each region has predetermined precise location information. Therefore, the acquisition unit 131 may acquire the precise location information of each of the base stations 30 (step S701). For example, the acquisition unit 131 may acquire the precise location information of each of the base stations 30 from the accession core system 50.

[0108] Furthermore, the acquisition unit 131 may acquire an installation drawing showing information about the installation location of each reference station 30 (step S702). It is preferable that the reference station 30 be installed in a location where there are no obstacles that create blind spots and radio waves can be easily received. For this reason, the reference station 30 is often installed on the roof of a building or the like. For this reason, the installation drawing may include, for example, structural information of the structure ST on which the reference station 30 is installed. For example, if the structure ST is a cubic building, the structural information may include information on the length, width, and height of the structure ST. The structure ST may be an example of an object OB located around the reference station 30.

[0109] The assignment unit 136 may calculate the location information LI1 of the structure ST based on the accurate location information of the reference station 30 and the installation drawing (step S703). For example, the assignment unit 136 may calculate the location information LI1 of the structure ST on which the reference station 30 is installed and link the calculated location information LI1 to the installation drawing as a variable. Specifically, the assignment unit 136 may calculate the location information LI1 of the structure ST as first location information based on the accurate location information of the reference station 30.

[0110] (6-2. Preprocessing (2)) Figure 8 is a flowchart showing the procedure for preprocessing PR2. Preprocessing PR2 may be a process that adds first position information to the image portion of object OB in the reference station image BG.

[0111] The acquisition unit 131 may acquire a reference station image BS (step S801). Multiple reference station image BS may exist depending on the area where the reference station 30 is installed. Multiple reference station image BS may exist depending on the observation range of the SAR satellite 10. In addition, the reference station image BS may be a full-color optical satellite image taken from above the ground L.

[0112] The assignment unit 136 may determine whether or not there are any reference station images BG to which location information has not been assigned (step S802).

[0113] If there is a reference station image BG for which location information has not been assigned (step S802; Yes), the assignment unit 136 may extract one reference station image BG for which location information has not been assigned and identify the structure ST included in the extracted reference station image BG (step S803). The structure ST may be an object OB (an object OB located around the reference station 30) on which the reference station 30 is installed, and the structure ST can be identified based on the installation drawing corresponding to the reference station 30.

[0114] Next, the assignment unit 136 may assign location information LI1 of the structure ST (structure ST identified in step S803) to the image portion of the extracted unassigned reference station image BG (step S804). The location information LI1 assigned here corresponds to first location information (absolute coordinates) based on the precise location information of the reference station 30 included in the reference station image BG.

[0115] Furthermore, the assignment unit 136 may terminate the process if there are no items for which location information has not yet been assigned (step S802; No).

[0116] Here, Figure 9 shows a specific example in which location information LI1 is added to the base station image BG by the preprocessing (PR1, PR2) described in Figures 7 and 8. In Figure 9, a specific example of adding location information LI1 is explained using an enlarged portion of the base station image BG1 shown in Figure 6(b). The base station image BG1 shown in Figure 9 includes a base station 31 (an example of a base station 30) and a structure ST1 on which the base station 31 is installed (an example of a structure ST located around the base station 30).

[0117] Furthermore, as shown in the example in Figure 9, the precise location information of the reference station 31 is assumed to be absolute coordinates (X0, Y0, Z0). In this state, the assignment unit 136 may calculate the location information LI1 of the structure ST1 in step S703 based on the absolute coordinates (X0, Y0, Z0) and an installation drawing that includes the structural information of the structure ST1. For example, the assignment unit 136 may calculate the location information LI1 of the structure ST1 based on the absolute coordinates (X0, Y0, Z0) and the information of the length, width, and height of the structure ST.

[0118] Here, in step S802, the assignment unit 136 extracts the base station image BG1 as a base station image BG without assigned location information. In step S803, the assignment unit 136 determines that the structure ST on which the base station 31 included in the base station image BG1 is installed is structure ST1.

[0119] As a result, in step S804, the assignment unit 136 assigns the position information LI1 of structure ST1 to the image portion of the reference station image BG1. Figure 9 shows an example in which the assignment unit 136 assigns absolute coordinates (X1, Y1, Z1), absolute coordinates (X2, Y2, Z2), absolute coordinates (X3, Y3, Z3), and absolute coordinates (X4, Y4, Z4) as the position information LI1 of structure ST1, based on the absolute coordinates (X0, Y0, Z0) of the reference station 31.

[0120] Figure 9 shows an example in which the assignment unit 136 assigns the location information LI1 of the structure ST1 on which the base station 31 is installed to the base station image BG1.

[0121] However, the reference station image BG1 includes various objects OB (for example, other buildings, parking lots, roads, etc.) in addition to the structure ST1. Therefore, the assignment unit 136 may also assign location information LI1 to objects OB other than the structure ST1 on which the reference station 31 is installed. For example, the assignment unit 136 can calculate the location information LI1 of each object OB around the structure ST1 based on the positional relationship between the structure ST1 on which the reference station 31 is installed and the objects OB around the structure ST1 (which can be determined from the installation drawing), and the location information LI1 of the structure ST1.

[0122] Furthermore, Figure 9 shows an example in which a single reference station image BG1 is targeted, and location information LI1 (first location information based on the precise location information of the reference station 30) of an object OB contained in the reference station image BG1 is added.

[0123] However, the assignment unit 136 may combine multiple reference station image BGs and assign the location information LI1 of the object OB contained in each of the multiple reference station image BGs to each of the multiple reference station image BGs. For example, the assignment unit 136 may calculate the location information LI1 of the structure ST contained in each reference station image BG based on the accurate location information of the reference station 30 contained in each reference station image BG in which at least a portion of area AR1 overlaps. Then, the assignment unit 136 may further calculate the location information LI1 of the object OB existing between the structure STs based on the location information LI1 calculated for each structure ST contained in each reference station image BG.

[0124] The following describes the operation procedures of the information processing device 100 during the absolute coordinate assignment process and the 3D conversion process, respectively. As shown in Figure 6(a), the SAR satellite 10 may orbit the Earth along its orbit TR, observe the same observation range at predetermined intervals, and transmit the SAR image SR of the observation results to the information processing device 100 each time it is captured. For this reason, the information processing device 100 may sequentially acquire SAR images SG for each observation range by sequential SAR observations corresponding to the orbit of the SAR satellite 10. The process described below is assumed to be for range R1 (Figure 6(a)) of the observation range, but similar processes can be performed for other observation ranges as well.

[0125] Furthermore, the absolute coordinate assignment process and the 3D conversion process may be performed in real time according to the orbit of the SAR satellite 10.

[0126] (6-3. Absolute Coordinate Assignment Process) In the absolute coordinate assignment process, a pseudo-optical image CL_SG is generated by colorizing the grayscale SAR image SG, and then absolute coordinates are assigned to the pseudo-optical image CL_SG by feature point matching. The operation procedures of the information processing device 100 in the colorization process in the absolute coordinate assignment process and the feature point matching in the paired coordinate assignment process will be explained below.

[0127] (6-3-1. Colorization Process) Figure 10 is a flowchart showing the procedure for colorization. The acquisition unit 131 may determine whether or not it was able to acquire the SAR image SG from the SAR satellite 10 (step S1001). For example, the acquisition unit 131 may determine whether or not it was able to acquire the SAR image SG1 in which the range R1 was observed by the SAR satellite 10 from the SAR satellite 10.

[0128] Since the SAR satellite 10 can sequentially acquire SAR images SG1 in which the range R1 has been photographed at a pace of 6 images per hour (144 images per 24 hours), the acquisition unit 131 can determine that it has acquired SAR images SG1 in which the range R1 has been observed from the SAR satellite 10 in accordance with the sequential acquisition at this pace.

[0129] The acquisition unit 131 may wait until it can acquire the SAR image SG1 in which the range R1 is observed from the SAR satellite 10 (step S1001; No).

[0130] On the other hand, if the first generation unit 133 has acquired a SAR image SG1 in which the range R1 has been observed from the SAR satellite 10 (step S1001; Yes), it may input the SAR image SG1 acquired this time (SAR image SG1 obtained by observing the range R1) into model M1 (step S1002). The SAR image SG1 acquired this time is the original SAR image SG1 (SAR image SG1 to which absolute coordinates are to be assigned) which has not been colorized and does not have absolute coordinates.

[0131] Then, the first generation unit 133 may generate a pseudo-optical image CL_SG1 in which an optical image corresponding to the original SAR image SG1 is pseudo-reproduced based on the output result of model M1 (step S1003).

[0132] (6-3-2. Feature Point Matching) Figure 11 is a flowchart of the feature point matching procedure. The acquisition unit 131 may determine whether or not a pseudo-optical image CL_SG has been generated (step S1101). If the pseudo-optical image CL_SG has not been generated (step S1101; No), the acquisition unit 131 may wait until the pseudo-optical image CL_SG is generated.

[0133] On the other hand, if the acquisition unit 131 has generated a pseudo-optical image CL_SG (step S1101; Yes), it may acquire the generated pseudo-optical image CL_SG (step S1102). As shown in Figure 10, if the first generation unit 133 has generated a pseudo-optical image CL_SG1, the acquisition unit 131 may determine in step S1102 that a pseudo-optical image CL_SG has been generated and acquire the pseudo-optical image CL_SG1 as the generated pseudo-optical image CL_SG. The following process will be explained using the pseudo-optical image CL_SG1 as an example of a pseudo-optical image CL_SG acquired in step S1102.

[0134] The extraction unit 134 may determine the information of area AR2 indicated by the pseudo-optical image CL_SG1 (step S1103). For example, the extraction unit 134 may determine the information of area AR2 indicated by the pseudo-optical image CL_SG1 based on the approximate positional information contained in the pseudo-optical image CL_SG1. For example, the extraction unit 134 may determine that area AR2 indicated by the pseudo-optical image CL_SG1 is data resulting from observation of range R1.

[0135] Therefore, the extraction unit 134 may determine whether or not the reference station 30 is included in area AR2 based on the information of area AR2 (step S1104). The information processing device 100 may be managed by the operator that installed the reference station 30, and may manage installation location information in a database that defines what kind of reference station 30 is installed at what location in each area. For this reason, the extraction unit 134 can determine whether or not the reference station 30 is included in area AR2 by comparing the information of area AR2 with the installation location information.

[0136] If the reference station 30 is not included in area AR2 (step S1104; No), the extraction unit 134 may return to step S1101 to enable feature point matching for other pseudo-optical images CL_SG that include the reference station 30.

[0137] On the other hand, if the reference station 30 is included in area AR2 (step S1104; Yes), the extraction unit 134 may acquire a reference station image BG corresponding to the pseudo-optical image CL_SG1 based on the information of area AR2 (step S1105). Since area AR2 indicated by the pseudo-optical image CL_SG1 is data resulting from observation of range R1, the extraction unit 134 may acquire a reference station image BG in which at least a part of range R1 has been photographed. This reference station image BG is image data to which first position information (position information LI1) has been added by pre-processing (PR1, PR2). In the following description, it will be assumed that the extraction unit 134 has acquired a reference station image BG1 as a reference station image BG in which at least a part of range R1 has been photographed.

[0138] The extraction unit 134 may use model M2 to extract feature points F1 in area AR1 shown by the reference station image BG1, and also extract feature points F2 in area AR2 shown by the pseudo-optical image CL_SG1 (step S1106).

[0139] The matching unit 135 may then perform matching of common feature points F between the reference station image BG1 and the pseudo-optical image CL_SG1 based on feature points F1 and F2 (step S1107). Specifically, the matching unit 135 may calculate a norm as the feature quantity of feature point F1 and feature point F2, and match the feature point F that minimizes the distance between the reference station image BG1 and the pseudo-optical image CL_SG1, i.e., between feature point F1 and feature point F2, as the common feature point F.

[0140] The assignment unit 136 may assign location information LI2 to the pseudo-optical image CL_SG1 based on the location information LI1, the feature point F2 that matches the feature point F1 that points to the object OB (e.g., structure ST1) for which location information LI1 has been calculated (step S1108). For example, the assignment unit 136 may assign location information LI2 to the pseudo-optical image CL_SG1 based on the feature point pair that matches between feature point F1 and feature point F2, the feature point F2 that matches the feature point F1 that points to the object OB, and the location information LI1 that has been assigned to the image portion of the object OB in the reference station image BG1. For example, the assignment unit 136 may perform information transfer to transfer the second location information to the image portions of all feature point F2 included in the pair in the pseudo-optical image CL_SG1. The location information LI2 assigned here is the second location information corresponding to the location information LI1 (first location information), and therefore can be said to be absolute location information based on the accurate location information of the base station 30.

[0141] As a result, the acquisition unit 131 acquires a pseudo-optical image CL_SG1' to which position information LI2 is attached (step S1109). The pseudo-optical image CL_SG1' to which position information LI2 is attached is image data in which the original SAR image SG1 has been colorized and absolute coordinates have been attached.

[0142] (6-4. 3D Processing) Figure 12 is a flowchart (1) showing the procedure for 3D processing. The 3D processing may be performed in real time according to the orbit of the SAR satellite 10, following the absolute coordinate assignment processing shown in Figure 11. Figure 12 shows the scene in which SD processing is performed based on the pseudo-optical image CL_SG1' to which absolute coordinates have been assigned.

[0143] The second generation unit 137 may acquire the original SAR image SG corresponding to the pseudo-optical image CL_SG1' to which absolute coordinates have been assigned (step S1201). Specifically, the second generation unit 137 may acquire the original SAR image SG1.

[0144] For example, the second generation unit 137 may input the original SAR image SG1 to the model M3 (step S1202a). Then, based on the output result output by the model M3, the second generation unit 137 may detect the region of object OB in area AR2 indicated by the original SAR image SG1 (step S1203a) and obtain object region information OB_DA2, which is information about the detected region (step S1204a).

[0145] Furthermore, the second generation unit 137 may input the original SAR image SG1 to the model M4 (step S1202b). Then, based on the output result output by the model M4, the second generation unit 137 may estimate the height of object OB in area AR2 indicated by the original SAR image SG1 (step S1203b) and obtain object height information OB_DA3, which is information about the estimated height (step S1204b).

[0146] The second generation unit 137 may then generate object information OB_DA, which includes object region information OB_DA2 and object height information OB_DA3 (step S1205).

[0147] Furthermore, the second generation unit 137 may synthesize an original SAR image OB_SG1 having object information OB_DA by associating object information OB with the original SAR image SG1 (step S1206). The original SAR image OB_SG1 has object information OB_DA but does not have absolute coordinates.

[0148] Figure 13 illustrates the 3D conversion process that follows the process shown in Figure 12. Figure 13 is a flowchart (2) showing the steps of the 3D conversion process.

[0149] The acquisition unit 131 acquires a pseudo-optical image CL_SG1' to which absolute coordinates have been assigned (step S1301).

[0150] The extraction unit 134 extracts feature points F in area AR2 indicated by the original SAR image OB_SG1 and in area AR2 indicated by the pseudo-optical image CL_SG'. Specifically, the extraction unit 134 uses model M2 to extract feature point F3, which is a feature point on the original SAR image OB_SG1 side in area AR2, and also extracts feature point F4, which is a feature point on the pseudo-optical image CL_SG1' side in area AR2 (step S1302).

[0151] The matching unit 135 may then perform a matching of common feature points F between the original SAR image OB_SG1 and the pseudo-optical image CL_SG1' based on feature points F3 and F4 (step S1303). Specifically, the matching unit 135 may calculate a norm as the feature quantity of feature point F3 and feature point F4, and match the feature point F that minimizes the distance between the original SAR image OB_SG1 and the pseudo-optical image CL_SG1', i.e., between feature point F3 and feature point F4, as the common feature point F.

[0152] The original SAR image OB_SG1 is grayscale, while the pseudo-optical image CL_SG1' is color. However, since both are SAR images with the same imaging principle, they can be appropriately matched.

[0153] The assignment unit 136 may assign object information OB_DA to the pseudo-optical image CL_SG1' based on the matching feature point pair between feature point F3 and feature point F4, and the object information OB_DA associated with the image portion of the paired feature point F3 in the original SAR image OB_SG1 (step S1304). For example, the assignment unit 136 may perform an information transfer to transfer the object information OB_DA to all image portions of the feature point F4 included in the pair in the pseudo-optical image CL_SG1'.

[0154] As a result, the acquisition unit 131 acquires a pseudo-optical image CL_SG1'' to which object information OB_DA is attached (step S1305). The pseudo-optical image CL_SG1'' to which object information OB_DA is attached is image data in which the original SAR image SG1 is colorized, absolute coordinates are attached, and information that forms the basis for 3D conversion is attached.

[0155] In this state, the second generation unit 137 may generate a three-dimensional model MD in which the object OB, which is represented in a plane in the pseudo-optical image CL_SG1'', is represented three-dimensionally based on the object information OB_DA assigned to the pseudo-optical image CL_SG1'' (step S1306). The second generation unit 137 may then generate the final three-dimensional model MD'' by processing the three-dimensional model MD to bring it closer to the actual colors according to the RGB elements included in the pseudo-optical image CL_SG1''.

[0156] (6-5. Services Utilizing Three-Dimensional Models) As a service utilizing three-dimensional models, the operating procedure of the information processing device 100 in support processing to deal with landslides will be described. Support processing may include calculation processing to calculate the volume of sediment and control processing for sediment removal, and the operating procedure of the information processing device 100 in each of these processes will be described below.

[0157] (6-5-1. Calculation Process) Figure 14 is a flowchart showing the procedure for calculating the volume of sediment. The acquisition unit 131 may, if a critical natural phenomenon (e.g., an earthquake) occurs while the three-dimensional model MD is being generated, acquire 3D model MD1 and 3D model MD2 as three-dimensional model MD before and after the timing when the occurrence of the critical natural phenomenon was detected (step S1401).

[0158] 3D model MD1 may be a 3D model MD of the ground surface of a designated area under normal conditions before a critical natural phenomenon occurs. Similarly, 3D model MD2 may be a 3D model MD of the ground surface of the same area after a critical natural phenomenon occurs.

[0159] The matching unit 135 may match the relative positional relationship between 3D model MD1 and 3D model MD2 (step S1402).

[0160] The estimation unit 138 may calculate the difference between the topography before the occurrence of the critical natural phenomenon and the topography after the occurrence of the critical natural phenomenon based on the matching results (step S1403).

[0161] Furthermore, the estimation unit 138 may detect the sediment portion based on the difference calculated in step S1403 (step S1404). For example, the estimation unit 138 may detect the portion corresponding to the difference (i.e., the portion that has changed between the topography before the occurrence of the critical natural phenomenon and the topography after the occurrence of the critical natural phenomenon) as the sediment portion.

[0162] Then, the estimation unit 138 may calculate the volume of soil corresponding to the soil portion based on the three-dimensional model MD corresponding to the soil portion (step S1405).

[0163] Furthermore, the estimation unit 138 may further detect whether or not a landslide has occurred based on the calculated volume of sediment. For example, the estimation unit 138 may determine whether or not the volume of sediment calculated in step S1405 exceeds a threshold (step S1406). If the volume of sediment does not exceed the threshold (step S1406; No), the process may be terminated.

[0164] On the other hand, if the volume of sediment calculated in step S1405 exceeds a threshold (step S1406; Yes), the estimation unit 138 may detect that a landslide has occurred.

[0165] (6-5-2. Control Processing for Soil Removal) Figure 15 is a flowchart showing the procedure for control processing for soil removal. When the estimation unit 138 detects that a soil disaster has occurred, it may estimate the type and number of equipment necessary for removing the soil based on the volume of the soil (step S1501).

[0166] The notification unit 139 may be controlled to notify the user U of information based on the estimation result (step S1502).

[0167] <Modification> The information processing according to the embodiment may be carried out in a manner different from the example above. Therefore, modifications of the information processing according to the embodiment will be described below.

[0168] In the above embodiment, the information processing device 100 pre-assigns first position information, which is absolute position information (absolute coordinates) based on the precise position information of the reference station 30, to the reference station image BG (optical image), and performs feature point matching with a pseudo-optical image CL_SG that includes the reference station 30, thereby assigning second position information corresponding to the first position information to the pseudo-optical image CL_SG. This method requires pre-processing to assign first position information to the reference station image BG, and the pseudo-optical image CL_SG used for feature point matching must include the reference station 30.

[0169] Therefore, as a modification, we will describe a method that eliminates the need for pre-processing to add first position information to the reference station image BG, and enables feature point matching even with a pseudo-optical image CL_SG that does not include the reference station 30. Below, we will describe in detail the information processing related to the modification.

[0170] [1. Overall Picture of the Modified Example] First, we will explain the overall picture of the modified example using Figure 16. Figure 16 is a diagram showing the overall picture of the modified example. Figure 16 corresponds to the example in Figure 2, but some parts are different due to the information processing related to the modified example. We will explain the differences in the information processing related to the modified example compared to the information processing according to one embodiment shown in Figure 2.

[0171] Figure 2 shows an example of an absolute coordinate assignment process in which first position information is assigned to the reference station image BG based on the precise position information of the reference station 30. Specifically, in the absolute coordinate assignment process, the position information of an object located around the reference station 30 and included in area AR1 is calculated as the first position information, based on the precise position information of the reference station 30, and an example is shown in which the first position information is assigned to the image portion of the object in the reference station image BG.

[0172] In this type of absolute coordinate assignment process, pre-processing is required to assign first position information to the reference station image BG, and the pseudo-optical image CL_SG used for feature point matching must include the reference station 30. In other words, the original SAR image SG must include the reference station 30.

[0173] Therefore, the inventors of the present invention have arrived at a method that eliminates the need for pre-processing to assign first position information to the reference station image BG, and enables feature point matching even with a pseudo-optical image CL_SG that does not include the reference station 30, by using an aerial photograph AP taken by an aircraft 70 with RTK positioning capabilities instead of the reference station image BG.

[0174] In the example shown in Figure 16, the information processing system Sy includes an aircraft 70 instead of an optical satellite 60. The aircraft 70 may be, for example, an unmanned aircraft such as a drone, or a small airplane piloted by a person.

[0175] The aircraft 70 may have a calculation function to calculate its own precise position information by RTK positioning, and a photographic function. The calculation function here may be a receiving device 20. That is, the aircraft 70 is equipped with a receiving device 20 and acquires aerial photographs AP by taking pictures with a camera from above.

[0176] For this reason, the aerial photograph AP has accurate positional information (absolute coordinates). Specifically, the aerial photograph AP has positional information based on the accurate positional information of the aircraft 70, and the positional information in area AR1 shown by the aerial photograph AP is assigned as first positional information at the shooting stage. For example, the shooting function of the aircraft 70 may, in cooperation with the receiving device 20, acquire an aerial photograph AP to which the positional information in area AR1 shown by the aerial photograph AP is assigned as first positional information, with positional information based on the accurate positional information of the aircraft 70.

[0177] Furthermore, the aerial photograph AP may be taken in advance, for example, by having the aircraft 70 fly over the entire country of Japan at any given time, and then uploaded to the information processing device 100. However, since the aerial photograph AP belongs to the category of full-color optical images, it may not be possible to accurately match feature points with the SAR image.

[0178] Therefore, in the absolute coordinate assignment process, which is performed in real time according to the orbit of the SAR satellite 10, the SAR image SG is colorized to generate a pseudo-optical image CL_SG, which is a full-color image (i.e., an RGB image) that closely resembles the color tones of an optical image. Furthermore, in the absolute coordinate assignment process, feature point matching between the pseudo-optical image CL_SG and the aerial photograph AP assigns a second position information corresponding to the first position information to the pseudo-optical image CL_SG, which does not have absolute coordinates. As a result, a pseudo-optical image CL_SG' is obtained. The second position information referred to here can be said to be absolute position information (absolute coordinates) based on the precise position information of the aircraft 70.

[0179] [2. Feature Point Matching Related to Modified Examples] Next, the specific operation procedure of the information processing device 100 in feature point matching related to modified examples will be explained. Figure 17 is a flowchart showing the procedure for feature point matching related to modified examples. In the example of Figure 17, it is assumed that the information processing device 100 has acquired the original SAR image SG1, which is the SAR image SG1 in which the range R1 was observed by the SAR satellite 10, and has already performed colorization of the original SAR image SG1 using Model M1. That is, in the example of Figure 17, it is assumed that the information processing device 100 has already generated a pseudo-optical image CL_SG1 in which an optical image corresponding to the original SAR image SG1 is pseudo-reproduced.

[0180] First, the acquisition unit 131 may determine whether or not a pseudo-optical image CL_SG has been generated (step S1701). If the pseudo-optical image CL_SG has not been generated (step S1701; No), the acquisition unit 131 may wait until a pseudo-optical image CL_SG is generated.

[0181] On the other hand, if the acquisition unit 131 has generated a pseudo-optical image CL_SG (step S1701; Yes), it may acquire the generated pseudo-optical image CL_SG (step S1702). Since the first generation unit 133 has already generated the pseudo-optical image CL_SG1, in step S1702, the acquisition unit 131 determines that the pseudo-optical image CL_SG has been generated and may acquire the pseudo-optical image CL_SG1 as the generated pseudo-optical image CL_SG. The following process will be explained using the pseudo-optical image CL_SG1 as an example of the pseudo-optical image CL_SG acquired in step S1702.

[0182] The extraction unit 134 may determine the information of area AR2 indicated by the pseudo-optical image CL_SG1 (step S1703). For example, the extraction unit 134 may determine the information of area AR2 indicated by the pseudo-optical image CL_SG1 based on the approximate positional information contained in the pseudo-optical image CL_SG1. For example, the extraction unit 134 may determine that area AR2 indicated by the pseudo-optical image CL_SG1 is data resulting from observation of range R1.

[0183] The extraction unit 134 may acquire an aerial photograph AP corresponding to the pseudo-optical image CL_SG1 based on the information of area AR2 (step S1704). Since area AR2 indicated by the pseudo-optical image CL_SG1 is data resulting from observation of range R1, the extraction unit 134 may acquire an aerial photograph AP in which at least a part of range R1 has been photographed. In the following description, it will be assumed that the extraction unit 134 has acquired an aerial photograph AP1 as an aerial photograph AP in which at least a part of range R1 has been photographed.

[0184] The extraction unit 134 may synchronize the coordinate systems between the aerial photograph AP1 and the pseudo-optical image CL_SG1 (step S1705). For example, the extraction unit 134 may perform a geographic coordinate transformation to synchronize the coordinate systems between the aerial photograph AP1 and the pseudo-optical image CL_SG1.

[0185] Furthermore, the extraction unit 134 may match the image scale between the aerial photograph AP1 and the pseudo-optical image CL_SG1 (step S1706). For example, since the flight altitude of the aircraft 70 and the SAR satellite 10 are different, the image scales of the aerial photograph AP1 and the pseudo-optical image CL_SG1 are different. For this reason, the extraction unit 134 may match the image scale by, for example, downsizing the resolution of the aerial photograph AP1.

[0186] In this state, the extraction unit 134 may use model M2 to extract feature points F1 in area AR1 shown in aerial photograph AP1, and also extract feature points F2 in area AR2 shown in pseudo-optical image CL_SG1 (step S1707).

[0187] The matching unit 135 may then perform matching of common feature points F between the aerial photograph AP1 and the pseudo-optical image CL_SG1 based on feature points F1 and F2 (step S1708). Specifically, the matching unit 135 may calculate a norm as the feature quantity of feature point F1 and feature point F2, and match the feature point F that minimizes the distance between the aerial photograph AP1 and the pseudo-optical image CL_SG1, i.e., between feature point F1 and feature point F2, as the common feature point F.

[0188] The assignment unit 136 may assign position information LI2 to the pseudo-optical image CL_SG based on the matching feature point pair between feature point F1 and feature point F2, and the position information LI1 assigned to the image portion of the paired feature point F1 in the aerial photograph AP1 (step S1709). For example, the assignment unit 136 may also transfer the position information LI2 on a pixel-by-pixel basis using the position information LI2 assigned to the image portion of all feature points F2 included in the pair in the pseudo-optical image CL_SG1.

[0189] As a result, the acquisition unit 131 acquires a pseudo-optical image CL_SG1' to which position information LI2 is attached (step S1710). The pseudo-optical image CL_SG1' to which position information LI2 is attached is image data in which the original SAR image SG1 has been colorized and absolute coordinates have been attached.

[0190] Thus, in the information processing related to the modified form, the pre-processing of assigning first position information is unnecessary, and feature point matching can be performed even with a pseudo-optical image CL_SG that does not include the reference station 30, thus enabling more efficient feature point matching.

[0191] <Hardware Configuration> The information processing device 100 described above may be implemented by a computer 1000 having a configuration such as that shown in Figure 18. Figure 18 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device 100. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

[0192] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0193] The HDD 1400 stores programs executed by the CPU 1100, and data used by such programs. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0194] The CPU 1100 controls output devices such as displays and input devices such as keyboards via the input / output interface 1600. The CPU 1100 acquires data from input devices via the input / output interface 1600. The CPU 1100 also outputs the generated data to output devices via the input / output interface 1600.

[0195] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0196] For example, when the computer 1000 functions as an information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded on the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from other devices via a predetermined communication network.

[0197] <Other> Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0198] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0199] Furthermore, the above embodiments can be combined as appropriate, provided that the processing content is not contradictory.

[0200] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, including the embodiments described in the section on the present invention.

[0201] Sy Information Processing System 1 SAR Observation System 2 RTK Positioning System 10 SAR Satellite 20 Receiving Device 30 Base Station 40 GNSS Satellite 50 Positioning Core System 60 Optical Satellite 70 Flight 100 Information Processing Device 120 Memory Unit 130 Control Unit 131 Acquisition Unit 132 Learning Unit 133 First Generation Unit 134 Extraction Unit 135 Matching Unit 136 Assignment Unit 137 Second Generation Unit 138 Estimation Unit 139 Notification Unit

Claims

1. An information processing system comprising: a first generation unit that generates an RGB image corresponding to a synthetic aperture radar image based on a predetermined synthetic aperture radar image and a predetermined machine learning model; an acquisition unit that acquires a predetermined optical image corresponding to the predetermined synthetic aperture radar image and to which first position information is assigned; an extraction unit that extracts feature points in a first area shown by the optical image and a second area shown by the RGB image; a matching unit that matches the feature points between the first area and the second area; and an assignment unit that assigns second position information corresponding to the first position information to the RGB image based on the matching result.

2. The information processing system according to claim 1, wherein the first generation unit generates an RGB image in which an optical image corresponding to the predetermined synthetic aperture radar image is pseudo-reproduced, by inputting the predetermined synthetic aperture radar image into a machine learning model that has been trained to pseudo-reproduce an optical image corresponding to the predetermined synthetic aperture radar image, using a combination of a sample SAR image and a sample optical image as training data, as a combination of images with similar shooting areas.

3. The information processing system according to claim 1, further comprising a reference station whose location information is known, wherein the acquisition unit acquires a satellite image as the optical image, which is an image including the reference station, and to which the first location information based on the location information of the reference station is assigned.

4. The information processing system according to claim 3, wherein the assignment unit assigns the first position information to the image portion of the object in the satellite image, by calculating the position information of an object located around the reference station that is included in the first area as first position information, based on the position information of the reference station.

5. The information processing system according to claim 4, wherein the extraction unit extracts first feature points which are feature points in the first area based on the satellite image and extracts second feature points which are feature points in the second area based on the RGB image; the matching unit matches the feature points common to the first feature points and the second feature points; and the assignment unit assigns the second location information to the RGB image by transferring absolute location information as second location information corresponding to the first location information to the entire second area, based on the second feature points which are matched with the first feature points which point to the object for which the first location information has been calculated, and the first location information.

6. The information processing system according to claim 1, further comprising an aircraft having a calculation function for calculating its own position information by RTK (Real Time Kinetic) positioning and a shooting function, wherein the acquisition unit acquires an aerial photograph as the optical image, which is an aerial photograph taken from above by the shooting function and to which the first position information is attached based on the position information of the aircraft calculated by the calculation function.

7. The information processing system according to claim 6, wherein the extraction unit extracts first feature points, which are feature points in the first area, based on the aerial photograph, and extracts second feature points, which are feature points in the second area, based on the RGB image; the matching unit matches the feature points common to the first feature points and the second feature points; and the assignment unit assigns the second location information to the RGB image by transferring absolute location information as second location information corresponding to the first location information to the entire second area, based on the second feature points matched with the first feature points and the first location information assigned to the image portion of the aerial photograph of the first feature points.

8. The information processing system according to claim 1, further comprising a second generation unit that generates a three-dimensional model having the second position information, which virtually represents the state of the second area, based on the predetermined synthetic aperture radar image and the RGB image to which the second position information is attached.

9. The information processing system according to claim 8, wherein the second generation unit detects the region of an object from the second area based on the predetermined synthetic aperture radar image and a predetermined machine learning model, estimates the height of the object indicated by the region, and synthesizes the predetermined synthetic aperture radar image having the object information by associating object information including at least the region and the height with the predetermined synthetic aperture radar image.

10. The information processing system according to claim 9, wherein the extraction unit extracts a third feature point, which is a feature point in the second area, based on the predetermined synthetic aperture radar image having the object information, and extracts a fourth feature point, which is a feature point in the second area, based on the RGB image to which the second position information is assigned; the matching unit matches the feature points common to the third feature point and the fourth feature point; and the assignment unit assigns the object information to the image portion of the fourth feature point in the RGB image to which the second position information is assigned, based on the fourth feature point matched with the third feature point and the object information associated with the image portion of the third feature point in the RGB image having the object information.

11. The information processing system according to claim 10, wherein the second generation unit generates the three-dimensional model from the RGB image to which the object information is attached.

12. An information processing method to be executed by an information processing system, comprising: a first generation step of generating an RGB image corresponding to a synthetic aperture radar image based on a predetermined synthetic aperture radar image and a predetermined machine learning model; an acquisition step of acquiring a predetermined optical image corresponding to the predetermined synthetic aperture radar image and to which first position information is assigned; an extraction step of extracting feature points in a first area shown by the optical image and a second area shown by the RGB image; a matching step of matching the feature points between the first area and the second area; and an assignment step of assigning second position information corresponding to the first position information to the RGB image based on the matching result.