System and method for processing and analyzing multipurpose image based on satellite image and remote sensing

Abstract

The present invention relates to a signal processing device for restoring an onboard SAR image, the signal processing device comprising: an RF transmission / reception unit that transmits / receives an analog signal to / from a target; a navigation sensor unit that senses navigation data of the signal processing device; a signal processing unit that converts the analog signal received by the RF transmission / reception unit into a digital signal; a memory unit; and a processor unit, wherein the memory unit is configured to store the digital signal after conversion by the signal processing unit and the sensed navigation data, and the processor unit is configured to generate an SAR image on the basis of the stored digital signal and navigation data.

Description

Satellite Imagery and Remote Sensing-based Multipurpose Image Processing and Analysis System and Method

[0001] The present invention relates to Synthetic Aperture Radar (SAR) technology, and more specifically, to a signal processing apparatus and method that are mounted on a small platform such as a satellite or drone and process SAR signals in real-time or near-real-time in an onboard environment to restore high-resolution images.

[0002] The present invention relates to the field of satellite image processing technology, and more specifically, to a satellite image 3D conversion device and method that generates a digital elevation model including elevation data of terrain and structures based on satellite images and performs 3D conversion based thereon.

[0003] The present invention relates to the field of environmental monitoring and remote sensing technology, and in particular to a technology for detecting algal blooms occurring within a body of water using satellite image data and quantitatively analyzing their intensity. More specifically, the present invention relates to an algal bloom detection system and method capable of efficiently detecting and managing algal blooms in a large body of water by performing a series of automated processing steps including preprocessing of satellite image data, deep learning-based segmentation, area classification using clustering techniques, and calculation and visualization of the intensity of the detected algal bloom areas.

[0004] Synthetic Aperture Radar (SAR) is a system capable of acquiring high-resolution radar images day and night, regardless of weather conditions. Unlike conventional radar systems, SAR utilizes the movement of aircraft or satellites to enable a single antenna to continuously receive signals from various locations. By processing the large volume of signal data generated through this process, it is possible to achieve an effect similar to that of a large antenna. This process yields high-resolution images, and a major advantage is the ability to generate high-precision images using equipment that is significantly smaller than the size of an actual physical antenna.

[0005] Furthermore, because SAR utilizes electromagnetic waves (microwaves), it enables ground observation even in cloudy, rainy, or dark environments. This offers the significant advantage of not being restricted by weather conditions, unlike optical cameras. Additionally, SAR can obtain high-resolution images through synthetic apertures generated from mobile platforms (satellites, aircraft) and allows for deeper analysis of surface characteristics or geological structures by using wavelengths longer than visible light. Moreover, SAR can obtain information on the distance and velocity of targets by utilizing the time delay and Doppler effect of the received signal. Furthermore, SAR systems employ various frequency bands, such as the X-band, L-band, and C-band; since each band possesses different characteristics, they can be utilized to suit a wide range of applications.

[0006] With the recent significant improvement in the resolution and quality of satellite data, technologies for generating digital elevation models (DSM) and 3D terrain models utilizing this data are being actively researched in various fields. Digital elevation models represent terrain elevation data in two dimensions, and the technology to convert this into a three-dimensional form to more accurately reproduce structural information of objects such as terrain, buildings, and roads can be utilized in various fields including construction, urban planning, environmental analysis, and disaster management.

[0007] 3D modeling technology based on satellite data focuses particularly on generating realistic, high-resolution models by combining optical imaging and SAR (radar) data. This technology is establishing itself as a tool to meet various needs, such as analyzing complex urban structures, natural topography, architectural design, and environmental changes.

[0008] Existing Digital Elevation Model (DSM)-based 3D modeling methods generally rely on manual work or use complex algorithms to perform data processing. However, these existing methods require a significant amount of time and computing resources during the 3D modeling process, resulting in high costs. Furthermore, manual processes such as noise removal, coordinate transformation, and texture mapping present a problem in that the quality of the results can vary depending on the operator's skill level.

[0009] To address these issues, there is a need for deep learning-based automated data processing technology. Deep learning models can efficiently process large-scale satellite data, precisely remove noise and outliers, and minimize data loss. Furthermore, they are suitable for generating high-resolution 3D models with precision and realism by learning the structures of complex terrains and buildings.

[0010] Algal blooms are one of the major indicators of water pollution and are emerging as a significant issue in terms of environmental and public safety management. Algal blooms primarily occur in water bodies rich in nutrients such as nitrogen and phosphorus, and can have serious impacts on aquatic ecosystems and human health. Therefore, technologies capable of early detection and efficient monitoring of algal blooms are essential for environmental protection and the establishment of sustainable management strategies.

[0011] With the recent advancement of satellite imagery technology, techniques capable of observing and analyzing algal blooms over wide areas are garnering attention. Satellite imagery is suitable for detecting algal blooms because it allows for real-time observation of vast regions, and it can be effectively utilized in large-scale bodies of water that are difficult to observe with the naked eye.

[0012] Existing technologies for detecting algal blooms using satellite imagery rely primarily on manual work or complex deep learning models. However, adapting to new regions or environmental conditions requires separate data learning and model retraining, which is inefficient in terms of time and cost.

[0013] Traditional image segmentation and clustering techniques are not suitable for detecting algal blooms over large areas and reveal limitations in detailed area detection. In particular, effective monitoring is difficult in areas with many islands or severe water level fluctuations due to low detection accuracy.

[0014] Most existing systems are operated manually or semi-automated, and their ability to detect algal blooms and analyze their intensity in real time is limited. This results in the disadvantage of being unable to respond immediately to environmental changes.

[0015] Conventional micro-SAR systems mounted on satellites and drones lack the technology to process received SAR signals in real-time or at near-real-time speeds. In particular, on miniaturized platforms, it has been difficult to integrate the high-performance hardware required to rapidly process large volumes of SAR data onto the board; consequently, SAR signal processing is often relied upon from external ground systems. In such cases, issues regarding latency and data transmission costs arise, making rapid, near-real-time SAR image reconstruction impossible.

[0016] This invention proposes a lightweight onboard SAR signal processing system for installation on satellites and aircraft, enabling high-performance signal processing within limited resources and allowing for rapid onboard processing of SAR data without relying on external systems. Through this, the invention aims to overcome the limitations of real-time signal processing in existing micro-SAR systems.

[0017] The present invention aims to solve the problem of useful data being lost during the noise and outlier removal process of existing digital elevation model (DSM)-based 3D modeling methods.

[0018] The present invention aims to solve the problem that 3D models generated by conventional methods often fail to properly represent complex terrain or the detailed structure of buildings.

[0019] The present invention aims to solve the problem that the existing DSM-based 3D modeling process relies mostly on manual work and traditional algorithms, resulting in a low level of automation.

[0020] The present invention provides a system capable of detecting algal blooms using satellite imagery data without prior training data. This reduces costs and time and enables efficient application in various regions and conditions.

[0021] The present invention aims to detect algal blooms with high accuracy even under environmental conditions such as diverse terrains like regions with many islands and water bodies with severe water level fluctuations, through a flexible machine learning-based algorithm.

[0022] The present invention aims to improve the precision of algal bloom detection by combining a deep learning-based segmentation technique with a K-means clustering algorithm. Through this, the present invention can detect algal bloom areas in detail within a body of water and analyze their intensity.

[0023] The present invention provides a system capable of automatically processing satellite imagery data and analyzing the status of algal blooms in real time. This enables the automation of the algal bloom detection and analysis process, allowing environmental management agencies to respond quickly.

[0024] The present invention visually represents the results of algal bloom detection and provides the intensity and distribution in the form of a heatmap or graph so that users can understand them intuitively. In addition, it aims to enhance practical usability by providing the results in a format that can be linked with GIS (Geographic Information System) data.

[0025] The present invention relates to a signal processing device for restoring onboard SAR images, comprising: an RF transceiver for transmitting and receiving an analog signal to and from a target; a navigation sensor for sensing navigation data of the signal processing device; a signal processing unit for converting the analog signal received by the RF transceiver into a digital signal; a memory unit; and a processor unit; wherein the memory unit is configured to store the digital signal converted by the signal processing unit and the sensed navigation data, and the processor unit is configured to generate a SAR image based on the stored digital signal and navigation data.

[0026] In addition, the processor unit is configured to perform signal processing operations of distance compression or directional compression on the digital signal stored in the memory unit.

[0027] Additionally, it further includes a data storage unit; said data storage unit is configured to permanently store digital signals and navigation data stored in said memory unit.

[0028] In addition, the signal processing unit further includes an ADC (Analog-to-Digital Converter) module, and the ADC module is configured to convert an analog signal received by the RF transceiver into a digital signal.

[0029] In addition, the memory unit is configured to store the converted digital signal and the sensed navigation data based on the same time.

[0030] In addition, the memory unit includes DDR memory, and the data storage unit is composed of an SSD and configured to temporarily and permanently store the digital signal and the navigation data.

[0031] The present invention relates to a signal processing device for restoring onboard SAR images, comprising: an RF transceiver for transmitting and receiving an analog signal to and from a target; a navigation sensor for sensing navigation data of the signal processing device; a signal processing unit for converting the analog signal received by the RF transceiver into a digital signal; a memory unit; and a processor unit; wherein the memory unit is configured to store the digital signal converted by the signal processing unit and the sensed navigation data, and the signal processing unit is configured to process the SAR signal by mounting a high-speed conversion algorithm specialized for the conversion of the SAR image onto a GPU device.

[0032] In addition, the processor unit is configured to restore a high-resolution SAR image based on the SAR signal processed by the signal processing unit.

[0033] In addition, the processor unit is configured to perform signal processing operations of distance compression or directional compression on the digital signal stored in the memory unit.

[0034] Additionally, it further includes a data storage unit; said data storage unit is configured to permanently store digital signals and navigation data stored in said memory unit.

[0035] In addition, the signal processing unit further includes an ADC (Analog-to-Digital Converter) module, and the ADC module is configured to convert an analog signal received by the RF transceiver into a digital signal.

[0036] In addition, the memory unit is configured to store the converted digital signal and the sensed navigation data based on the same time.

[0037] In addition, the memory unit includes DDR memory, and the data storage unit is composed of an SSD and configured to temporarily and permanently store the digital signal and the navigation data.

[0038] The present invention relates to a signal processing method for a signal processing device that restores an onboard SAR image, comprising: a step of transmitting and receiving an analog signal to a target; a step of sensing navigation data of the signal processing device; a step of converting the received analog signal into a digital signal; a step of storing the converted digital signal and the sensed navigation data; and a step of processing the converted digital signal by mounting a high-speed conversion algorithm specialized for the conversion of the SAR image on a GPU device.

[0039] The present invention relates to a method for generating a 3D model based on an image of a satellite image using a deep learning model, comprising: a step of generating a digital elevation model based on an image of the satellite image using the deep learning model; a point cloud generation step of generating a point cloud based on the generated digital elevation model; a removal step of removing at least one of noise and outliers from the generated point cloud; a step of generating a mesh based on the point cloud from which at least one of the noise and outliers has been removed; and a step of generating a 3D model based on the generated mesh.

[0040] In addition, the point cloud generation step further includes a pixel coordinate extraction step for extracting the X, Y, and Z values ​​of each pixel of the digital elevation model.

[0041] Additionally, the pixel coordinate extraction step further includes a Z-coordinate scaling step for reflecting the actual altitude of the satellite image with respect to the extracted Z value.

[0042] Additionally, the Z-coordinate scaling step further includes a step of removing outliers from the Z-value of the pixel or performing interpolation using surrounding data for missing values.

[0043] In addition, the point cloud generation step further includes the step of generating a reference point in 3D space by integrating the extracted X, Y, and Z coordinate data.

[0044] Additionally, the point cloud generation step further includes: a step of searching for neighbor points around the generated reference point; and a step of calculating a normal vector based on the generated reference point and the searched neighbor points.

[0045] Additionally, the above removal step further includes the step of removing at least one point among points having a density below a threshold and points located at a position above a threshold from the expected position for the generated point cloud.

[0046] Additionally, the above removal step further includes a step of analyzing the density and location distribution of points using at least one of the K-Nearest Neighbors (KNN) algorithm and the Density-Based Clustering (DBSCAN) technique for the generated point cloud.

[0047] Additionally, the removal step further includes a step of considering a point as an outlier if, for the generated point cloud, the attribute of at least one point among RGB values ​​and intensity values ​​has a difference greater than a threshold from surrounding points.

[0048] The present invention relates to a satellite image 3D conversion system that generates a 3D model based on an image of a satellite image using a deep learning model, comprising: a digital elevation model generation module that generates a digital elevation model based on an image of the satellite image using the deep learning model; a point cloud generation module that generates a point cloud based on the generated digital elevation model; a normal vector calculation module for generating a normal vector of each point based on the generated point cloud; a noise and outlier removal module for controlling at least one of the noise and outliers of the generated point cloud; a 3D mesh generation module that generates a 3D mesh based on the point cloud from which at least one of the noise and outliers has been removed; and a visualization module that generates a 3D model based on the generated 3D mesh.

[0049] In addition, the digital elevation model generation module is configured to correct the brightness or contrast of the satellite image by utilizing a Convolutional Neural Network (CNN) model.

[0050] In addition, the digital elevation model generation module is configured to predict pixel values ​​by learning information around damaged areas using a Convolutional Neural Network (CNN) model when there are missing pixels or damaged areas in the satellite image data.

[0051] In addition, the digital elevation model generation module is configured to align a plurality of images of the satellite imagery into the same coordinate system for stereo matching of the plurality of images, and to correct lighting and color differences between the plurality of images.

[0052] In addition, the digital elevation model generation module is configured to extract depth information by calculating the pixel displacement (disparity) of the same point between the plurality of images.

[0053] Additionally, it further includes a coordinate transformation and scaling module; wherein the coordinate transformation and scaling module is configured to perform a transformation into X, Y coordinates on a 2D plane based on row and column information of each pixel from the data of the generated digital elevation model, and to extract altitude information, which is the Z value of each pixel, from the data of the generated digital elevation model.

[0054] In addition, the coordinate transformation and scaling module is configured to generate a 3D coordinate array in the form of [X, Y, Z] using the transformed X, Y coordinates and the extracted Z value as inputs.

[0055] In addition, the coordinate transformation and scaling module is configured to generate the 3D coordinate array by utilizing a Multi-Layer Perceptron (MLP) model.

[0056] Additionally, the coordinate transformation and scaling module is configured to perform at least one of limiting the range of the extracted Z-coordinate value, adjusting the range of the extracted Z-coordinate value, removing outliers from the extracted Z-coordinate value, and performing supplementation using interpolation for missing values ​​of the extracted Z-coordinate value when the extracted Z-coordinate value deviates from the expected range.

[0057] In addition, the coordinate transformation and scaling module is configured to perform scaling on the extracted Z-coordinate value by utilizing a Recurrent Neural Network (RNN) model.

[0058] In addition, the normal vector calculation module is configured to calculate a reference point and neighbor points based on the generated point cloud, and to calculate a normal vector based on the relationship between the calculated reference point and neighbor points.

[0059] In addition, the normal vector calculation module is configured to generate a covariance matrix by centering the neighbor points of the reference point, and to determine the orientation of the virtual surface of the 3D object extracted from the generated point cloud by performing eigenvalue decomposition on the generated covariance matrix.

[0060] In addition, the noise and outlier removal module is configured to detect noise by utilizing at least one model among an Autoencoder model, a Graph Neural Network (GNN) model, and a Convolutional Neural Network (CNN) model.

[0061] The present invention relates to a green algae detection system for detecting green algae in a body of water using satellite image data, comprising: a satellite image input device for receiving the satellite image data; an image processing module for analyzing the received satellite image data and performing separation into R (red), G (green), and B (blue) channels; a segmentation module for separating sea areas and land areas by applying a deep learning-based algorithm based on the separated image data; and a clustering module configured to generate clusters for the separated sea areas.

[0062] In addition, the image processing module is configured to analyze the received satellite image data and separate the R (red) channel, G (green) channel, and B (blue) channel.

[0063] In addition, the image processing module is configured to distinguish a water area including non-water area and first water area data of the received satellite image data by utilizing at least one segmentation technique among a color-based filtering technique, a brightness-based filtering technique, and a texture-based filtering technique.

[0064] In addition, the segmentation module is configured to distinguish detailed regions within the water body for the first water body data by applying at least one deep learning-based algorithm among U-Net, Fully Convolutional Network, and DeepLab.

[0065] In addition, the segmentation module is configured to predict the class of each pixel of the image data containing the first water body data by utilizing a pre-trained deep learning model.

[0066] In addition, the segmentation module is configured to perform a prediction on whether each pixel of the image data containing the first water body data corresponds to an algal bloom area, a water area, or other areas by utilizing a pre-trained deep learning model.

[0067] In addition, the segmentation module is configured to classify pixels predicted as the algal bloom area into the generated second water body data.

[0068] In addition, the clustering module is configured to execute the K-Means Clustering algorithm to group the generated second body data into major color clusters.

[0069] Additionally, it further includes an algae detection mask generation module; wherein the clustering module is configured to select a cluster related to algae among the generated clusters, and the algae detection mask generation module is configured to perform binarization on the selected cluster.

[0070] In addition, the above-mentioned algae detection mask generation module is configured to set pixel values ​​belonging to the selected cluster to white and the remaining pixel values ​​not belonging to the selected cluster to black.

[0071] The present invention relates to a green algae detection system for detecting the intensity of a green algae area within a body of water using received satellite image data, comprising: a segmentation module configured to generate body of water data by applying a deep learning-based algorithm based on the satellite image data; a clustering module configured to generate clusters based on the generated body of water data and to select clusters related to green algae among the generated clusters; a green algae detection mask generation module configured to generate a detection mask based on the selected clusters; and an intensity calculation and visualization module configured to calculate the intensity of the green algae area based on the generated detection mask.

[0072] In addition, the above-mentioned algal bloom detection mask generation module is configured to perform binarization based on the selected cluster.

[0073] In addition, the above-described algal bloom detection mask generation module is configured to generate a detection mask composed of 0s and 1s based on the binarization performed above.

[0074] In addition, the intensity calculation and visualization module is configured to estimate the concentration of algal bloom based on at least one of the average and sum of the G channel values ​​of the generated detection mask.

[0075] In addition, the intensity calculation and visualization module is configured to determine the distribution and size of the algae by calculating the ratio of white pixels representing the algae area and black pixels representing the non-algae area in the generated detection mask.

[0076] In addition, the intensity calculation and visualization module is configured to quantify the severity of the green algae region based on pixel density or color intensity for the generated detection mask.

[0077] The present invention relates to a method for detecting algal blooms in a body of water using received satellite image data, comprising: a step of preprocessing the received satellite image data to extract first body of water data; a step of generating second body of water data by performing deep learning-based segmentation based on the extracted first body of water data; a step of generating clusters based on the generated second body of water data; and a step of generating an algal bloom area detection mask based on the generated clusters.

[0078] Additionally, the preprocessing step further comprises: a step of analyzing the received satellite image data to separate the R (red) channel, G (green) channel, and B (blue) channel; a step of performing correction by adjusting the balance of at least one of the brightness and color of each separated channel; and a step of integrating each channel for which correction has been performed.

[0079] Additionally, the extraction step further includes a step of distinguishing a water area including a non-water area of ​​the received satellite image data and the first water area data by utilizing at least one segmentation technique among a color-based filtering technique, a brightness-based filtering technique, and a texture-based filtering technique.

[0080] Additionally, the segmentation step further includes a step of distinguishing detailed regions within the water body for the extracted first water body data by applying at least one deep learning-based algorithm among U-Net, Fully Convolutional Network, and DeepLab.

[0081] In addition, the segmentation step further includes a prediction step that predicts the class of each pixel of image data containing the first body of water data using a pre-trained deep learning model.

[0082] In addition, the prediction step further includes a step of predicting whether each pixel of the image data containing the first water body data corresponds to an algal bloom area, a water area, or other areas by utilizing a pre-trained deep learning model.

[0083] Additionally, the clustering generation step further includes a step of selecting a cluster related to algal bloom among the generated clusters, and the detection mask generation step further includes a step of performing binarization based on the selected cluster.

[0084] The present invention enables high-speed SAR signal processing at near-real-time speeds. That is, through lightweight hardware including FPGA logic and GPU chips, the present invention enables SAR signal processing to be performed in real-time or at near-real-time speeds even in ultra-small SAR systems. This dramatically improves the latency issues of existing SAR systems, thereby enabling rapid SAR image restoration.

[0085] The present invention provides the effect of reducing data transmission delays and costs through onboard processing. That is, in conventional technology, received SAR data had to be transmitted to an external system for processing, but in the present invention, all SAR signal processing is performed onboard, thereby reducing data transmission delays and the associated costs. In particular, for satellite and drone environments, it can provide the effect of efficiently utilizing limited communication resources and bandwidth.

[0086] The present invention provides the benefits of lightweight and low-power design. Specifically, the present invention proposes a lightweight SAR signal processing system suitable for miniaturized satellites and drones, and enables stable signal processing even in power supply environments due to its low-power design. This makes it suitable for long-term satellite missions or use in small drones.

[0087] The present invention can provide the effect of high-resolution image restoration. Specifically, the present invention can provide a technology capable of accurately restoring high-resolution SAR images by combining navigation sensors and SAR signal data. This can further enhance the usability of SAR systems, particularly in various application fields such as terrain analysis, disaster monitoring, and military surveillance.

[0088] The present invention can provide the effect of reducing the time and cost required for data processing by automating the 3D modeling process using satellite data.

[0089] The present invention can rapidly process large-scale satellite data without complex manual work, thereby providing the effect of increasing the productivity of data utilization in various fields such as construction, environmental monitoring, and urban planning.

[0090] The present invention can provide the effect of increasing data accuracy through an automated noise removal and outlier correction process.

[0091] The present invention can provide the effect of generating a high-resolution, precise 3D model by applying deep learning-based mesh generation and texture mapping technology.

[0092] The present invention can provide the effect of utilizing high-resolution 3D terrain models in various fields such as urban planning, construction design, environmental analysis, and disaster management.

[0093] The present invention can provide the effect of enhancing usability in practical environments by precisely representing the complex structure of terrain and the detailed form of buildings.

[0094] The present invention can provide the effect of automating all steps of the 3D model creation process, such as coordinate transformation, point cloud generation, triangle mesh generation, texture mapping, and lighting settings, by utilizing a deep learning model.

[0095] The present invention can provide the effect of creating a realistic 3D model including high-resolution textures and lighting effects by combining optical images and SAR data extracted from satellite images.

[0096] The present invention can provide the effect of realizing natural visualization even under various environmental conditions through the lighting and material setting process.

[0097] The present invention detects algal blooms by analyzing satellite images without separate training data through a machine learning-based algorithm, thereby eliminating the need for large-scale training data preparation or high-performance hardware, and thus provides the effect of reducing costs and increasing the efficiency of system operation.

[0098] The present invention enables real-time detection of algal blooms based on satellite image data, thereby providing the effect of enabling fast and accurate environmental monitoring.

[0099] The present invention can be efficiently applied even in complex areas with many islands or frequent water level fluctuations, thereby providing the effect of stably detecting algal blooms regardless of various topography and environmental conditions.

[0100] The present invention provides the effect of enabling monitoring of large-scale water bodies by rapidly detecting algal blooms through a machine learning-based automated algorithm, thereby increasing the speed of analysis.

[0101] The present invention visualizes detected algal bloom data by intensity and distribution, providing users with intuitive information and the effect of clearly identifying the location of the algal bloom.

[0102] The present invention operates based on an automated algorithm, thereby minimizing manual intervention and simplifying the detection process to provide the effect of reducing human error.

[0103] FIG. 1 is a diagram illustrating a schematic sequence for SAR image reconstruction according to the present invention.

[0104] FIG. 2 is a diagram illustrating the configuration of a signal processing device according to the present invention.

[0105] Figure 3 is a diagram visualizing the signal processing process for SAR image reconstruction step by step.

[0106] Figure 4 is a diagram visualizing the processing steps of two major algorithms used for SAR image reconstruction: the Range-Doppler Algorithm (RDA) and the Back Projection Algorithm (BPA).

[0107] Figure 5 is a diagram showing how image data generated during the SAR image restoration process can be compared.

[0108] FIG. 6 is a diagram illustrating the configuration of a satellite image 3D conversion system according to an embodiment of the present invention.

[0109] FIG. 7 is a diagram showing the steps for performing a satellite image 3D conversion system according to an embodiment of the present invention.

[0110] FIG. 8 is a diagram illustrating the steps for generating a digital elevation model according to an embodiment of the present invention.

[0111] FIG. 9 is a diagram illustrating coordinate transformation and scaling steps according to an embodiment of the present invention.

[0112] FIG. 10 is a diagram illustrating the step of generating a point cloud and calculating a normal vector according to an embodiment of the present invention.

[0113] FIG. 11 is a diagram illustrating a noise and outlier removal step according to an embodiment of the present invention.

[0114] FIG. 12 is a diagram illustrating the steps for generating a 3D mesh and constructing a high-resolution 3D model according to an embodiment of the present invention.

[0115] FIG. 13 is a drawing showing a digital elevation model generated according to one embodiment of the present invention.

[0116] FIG. 14 is a drawing showing a converted point cloud model according to one embodiment of the present invention.

[0117] FIG. 15 is a drawing showing the final generated result according to one embodiment of the present invention.

[0118] FIG. 16 is a diagram showing the steps for performing an algal bloom detection system according to one embodiment of the present invention.

[0119] FIG. 17 is a diagram illustrating the configuration of a green algae detection system according to one embodiment of the present invention.

[0120] FIG. 18a is a diagram illustrating an embodiment for separating a detection target area using raw satellite image data according to the present invention.

[0121] FIG. 18b is a drawing for explaining an embodiment of calculating and visualizing algal bloom intensity by generating an algal bloom detection mask using actual images according to the present invention.

[0122] FIG. 19 is a drawing for explaining an embodiment of combining algal bloom intensity information with a real image using a real image according to the present invention.

[0123] Specific details of the embodiments are included in the detailed description and drawings.

[0124] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0125] The NanoSAR-B system tested for the present invention is a miniaturized SAR system weighing less than approximately 10 kg. Tests were conducted to acquire SAR data for drones by mounting it on a civilian aircraft and performing flight tests. This system uses a frequency-modulated continuous wave (FMCW) method to calculate the distance to a target based on the time difference between the transmitted and received signals, and can provide high-resolution images using the X-band frequency range of approximately 10.0 to 10.5 GHz. Since this system can maintain high resolution even at low altitudes, it has been confirmed that it is suitable for small platforms such as drones.

[0126] According to one embodiment of the present invention, the performance of representative image reconstruction algorithms, such as RDA (Range-Doppler Algorithm), CSA (Chirp Scaling Algorithm), and BPA (Back Projection Algorithm), can be compared using SAR data acquired through flight testing. Specifically, RDA (Range-Doppler Algorithm) is an algorithm based on range-Doppler transform, which can reconstruct an image by processing the SAR signal in the frequency domain.

[0127] RDA is an algorithm that can be used efficiently, particularly in environments with limited computing resources, and can be utilized in a very fast and general manner. Additionally, the Chirp Scaling Algorithm (CSA) is a method for reconstructing SAR images by transforming signals in the frequency domain. In other words, CSA is an improved version of RDA that enhances reconstruction performance through curve fitting during signal processing and can demonstrate good performance, especially in high-resolution SAR images. Furthermore, the Back Projection Algorithm (BPA) is a method of processing signals in the time domain that can generate an image by individually projecting all received signals. While BPA offers very high accuracy, it can have the disadvantage of high computational costs.

[0128] According to one embodiment of the present invention, the system of the experimental example may be a FW-CW-based conformal SAR weighing approximately 10 kg or less, and the data may be obtained by performing flight tests with NanoSAR-B mounted on a civilian aircraft rather than a drone to acquire SAR data for drones. Specifically, unlike conventional pulse radar, Frequency Modulated Continuous Wave (FMCW) can transmit continuous waveforms. The transmitted signal has the characteristic of changing frequency linearly over time, and through this, the distance to a target can be calculated using the time difference (delay time) with the received signal. Frequency modulation can generally be performed in the form of a triangular wave or a sawtooth wave. Unlike pulse radar, FW-CW transmits and receives simultaneously, and because it uses a continuous signal, it can measure distance and speed simultaneously. This can be advantageous for SAR systems that require accurate distance measurement over a short range.

[0129] According to one embodiment of the present invention, a FW-CW-based SAR system provides high frequency resolution, thereby enabling the reconstruction of high-resolution images. Furthermore, FW-CW is suitable for miniaturized systems and can perform SAR functions without complex pulse transmitters, making it suitable for ultra-small SAR platforms such as drones. In a SAR system for drones, FW-CW can provide the advantage of generating high-performance SAR images while reducing weight, power consumption, and size. Additionally, FW-CW generally exhibits excellent performance within a short range and can provide high accuracy, particularly in short-range detection. This makes it suitable for observing ground targets from a drone or for SAR systems operating at short altitudes.

[0130] According to one embodiment of the present invention, the nanoSAR-B system can use a frequency band of 10.0 to 10.5 GHz. This is a frequency band belonging to the X-band, which has short wavelengths and can provide high-resolution images. In particular, it may be suitable for detecting minute changes on the surface. Specifically, the X-band can generally be used in various application fields such as military, geological surveys, marine monitoring, and disaster response. It can operate at a low altitude of about 500 m.

[0131] According to one embodiment of the present invention, the nanoSAR-B system can use a maximum transmit power of 1 watt. This means that the SAR system is designed for low power consumption and can be used efficiently on small platforms such as drones. The low-power design allows for the transmission of reliable SAR signals while reducing the power consumption of the drone. Although transmit power affects signal quality depending on the distance to the target, system resolution, and environmental conditions, the nanoSAR-B optimizes these factors to maximize power efficiency.

[0132] According to one embodiment of the present invention, while a SAR system can generally be used at high altitudes, nanoSAR-B can be designed to generate high-resolution images even at low altitudes. A flight altitude of 500m is the range in which the drone primarily operates, allowing for the collection of more detailed SAR data close to the target. Operation at low altitudes has the advantage of enabling more precise detection and analysis of ground objects.

[0133] According to one embodiment of the present invention, a SAR system can provide a high resolution of up to 0.1 m. In a SAR system, resolution can be an important factor in determining how detailed an image can be obtained. A resolution of 0.1 m is a very high level, which means that even small objects on the ground can be clearly identified while the drone is flying. High-resolution images can play an important role in various applications such as terrain analysis, disaster monitoring, precision agriculture, and military surveillance.

[0134] According to one embodiment of the present invention, a flight test was performed by mounting NanoSAR-B on the aircraft PIPER-PA31.

[0135] According to the present invention, the nanoSAR-B system is lightweight to be suitable for small UAVs and can provide SAR capabilities without compromising the flight performance of the drone. Furthermore, through a low-power design, continuous SAR signal collection and transmission are possible while reducing the drone's battery consumption. Additionally, the nanoSAR-B can collect high-resolution SAR images in real-time or near-real-time at low altitudes, enabling rapid information provision in various application fields. Moreover, the SAR system can be flexibly integrated to suit various platforms, allowing for easy application for commercial or military purposes.

[0136] Figure 3 is a diagram visualizing the signal processing process for SAR image reconstruction step by step.

[0137] As described, Raw Data (41) represents the raw signal collected by the SAR system, in a state where preprocessing and compression have not yet been performed. That is, Raw Data (41) contains all received signals, resulting in a noisy and difficult-to-interpret state. Additionally, Range Compressed Data (42) represents the result of applying a Range Reference Function (44) to the raw data to compress it in the distance direction. In this process, processing can be performed to increase the resolution of the received signal in the distance direction. The Range Reference Function (44) strengthens the signal in the distance direction, enabling the acquisition of accurate location information of the target.

[0138] As described, Raw Data (41) represents the raw signal collected by the SAR system, in a state where preprocessing and compression have not yet been performed. That is, Raw Data (41) contains all received signals, resulting in a noisy and difficult-to-interpret state. Additionally, Range Compressed Data (42) represents the result of applying a Range Reference Function (44) to the raw data to compress it in the distance direction. In this process, processing can be performed to increase the resolution of the received signal in the distance direction. The Range Reference Function (44) strengthens the signal in the distance direction, enabling the acquisition of accurate location information of the target.

[0139] According to one embodiment of the present invention, Image Data (43) is final SAR image data, and by applying an Azimuth Reference Function (45) to Range Compressed Data (42) to perform compression in the azimuth direction as well, a high-resolution final image can be generated. The data of the generated high-resolution final image is the result of the SAR system detecting terrain, and it has a resolution level sufficient to identify targets. In addition, the reference functions of the Range Reference Function (44) and the Azimuth Reference Function (45) can each perform the role of compressing signals in the distance and azimuth directions, respectively. The Range Reference Function (44) amplifies the signal strength in the distance direction to increase resolution, and the Azimuth Reference Function (45) aligns the signal in the azimuth direction to enable the restoration of a clearer image.

[0140] Figure 4 is a diagram visualizing the processing steps of two major algorithms used for SAR image reconstruction: the Range-Doppler Algorithm (RDA) and the Back Projection Algorithm (BPA).

[0141] According to one embodiment of the present invention, the processing speed of GPU-based BPA image restoration can be analyzed by configuring a GPU kernel optimized for BPA signal processing. BPA is an image restoration algorithm widely used to generate high-resolution SAR images, and SAR data can be processed efficiently by utilizing the parallel processing performance of the GPU. This process includes several steps from raw data to the generation of the final SAR image, and iterative calculations can be performed using the kernel of the GPU.

[0142] The upper Range-Doppler Algorithm (RDA) processing steps illustrated in FIG. 4 are described below. The Raw Radar Data step (51), that is, the first step of the RDA, is to receive raw radar data. Additionally, the Range Compression step (52) is a step that performs range compression on the raw data. That is, the Range Compression step (52) can compress the radar signal to improve the resolution in the range direction.

[0143] According to one embodiment of the present invention, after distance compression, the signal in the azimuth direction can be processed in the frequency domain by performing a Fourier Transform (FFT) in the Azimuth FFT step (53). Subsequently, in the RCMC (Range Cell Migration Correction) step (54), the RCMC is a step for correcting so that each cell is maintained constant, and can correct signal distortion caused by the moving platform. Subsequently, the Azimuth Compression step (55) may be an azimuth compression step for improving the resolution in the azimuth direction, and in the Azimuth IFFT step (56), an Inverse Fourier Transform (IFFT) can be applied to convert the azimuth-compressed data back into the time domain.

[0144] Finally, in the Compressed Data step (57), the compressed data is finally obtained, and this data can be used to generate a high-resolution SAR image. That is, the RDA processing step is a fast and efficient method for processing SAR data, which requires less computation and can be used primarily in environments with limited computing resources.

[0145] The Back Projection Algorithm (BPA) processing step at the bottom, illustrated in FIG. 4, is described below. The Raw Radar Data step (51) can also begin by receiving raw radar data. Subsequently, in the Range Compression step (52), range compression can be performed to improve the range resolution, similar to RDA. Additionally, in the Flight Path Set-up step (58), it may be necessary to set the flight path of the aircraft. That is, in the Flight Path Set-up step (58), an accurate flight path can be set using GPS and IMU data, and preparations to correct SAR data based on this can begin.

[0146] Accordingly, the Continuous Motion Correction step (59) may be a step that improves the accuracy of the signal by correcting the movement of the platform that occurs during flight. Additionally, in the Azimuth Compression step (60), the signal in the azimuth direction may be compressed to improve the resolution. Accordingly, in the Compressed Data step (61), compressed data is finally generated, and this data can be used to generate a high-resolution SAR image. Consequently, it can be seen that BPA provides very high accuracy by individually projecting all received signals to generate an image, but has the disadvantage of slow processing speed due to the large amount of computation. BPA can be used to obtain high-resolution and precise SAR images.

[0147] According to the present invention, RDA can generate SAR images relatively quickly using an efficient algorithm, but its accuracy may be lower than that of BPA. On the other hand, BPA has high accuracy and is advantageous for analyzing complex terrain, but it may require significant computational resources due to its slow computation speed. Therefore, the present invention allows for the selection of an appropriate algorithm according to various requirements for SAR image reconstruction.

[0148] Figure 5 is a diagram showing how image data generated during the SAR image restoration process can be compared.

[0149] As described above, the characteristics of each image are related to the SAR data processing step, and the efficiency of the SAR image is explained below through comparison with the original optical image.

[0150] According to one embodiment of the present invention, FIG. 5(a) shows data compressed only in the distance direction. Since it has not yet been compressed in the azimuth direction, it can be seen that the resolution is low and it is difficult to identify specific shapes. Range Compressed Data is an initial processing stage of a SAR signal in which the resolution is improved only in the distance direction. Range Compressed Data focuses on precisely extracting distance information from raw radar data and does not include precise details in the azimuth direction.

[0151] According to one embodiment of the present invention, FIG. 5(b) is a SAR image generated by applying a Back Projection Algorithm (BPA) to compress data in both the azimuth and distance directions. Specifically, the data restored through BPA is a high-resolution SAR image that clearly reveals surface structures and details of the terrain. Since BPA generates an image by individually projecting all received signals onto each pixel, it can provide high accuracy. FIG. 5(b) represents the final stage of SAR data processing and exhibits a resolution and quality close to that of an optical image.

[0152] According to one embodiment of the present invention, FIG. 5(c) is an optical image of the same area of ​​the Optic Image, taken using a standard camera. While the Optic Image is affected by weather or illumination, the SAR Image can be less affected by such environmental factors because it uses radio waves.

[0153] In summary, Range Compressed Data (a) in Fig. 5 is the result of distance compression in the initial stage of SAR data processing, making it difficult to identify details, whereas BPA Processing Data (b) in Fig. 6 is a high-resolution restored SAR image that provides information similar to the actual terrain. Optic Image (c) in Fig. 5 is an image captured by an optical camera, and the accuracy of SAR image restoration can be evaluated by comparing it with the BPA result. In other words, it can be confirmed that BPA is a suitable algorithm for SAR image restoration. SAR images have environmental advantages that optical images lack, and in particular, they have the advantage of enabling terrain analysis regardless of the weather.

[0154] FIG. 1 is a diagram illustrating a schematic sequence for SAR image reconstruction according to the present invention. FIG. 2 is a diagram illustrating the configuration of a signal processing device according to the present invention.

[0155] As described above, the signal processing device (100) according to the present invention may include a signal processing unit (110), a memory unit (120), a data storage unit (130), and a processor unit (140). Specifically, the signal processing unit (110) may include an RF transceiver unit (111) and a navigation sensor unit (112). Below, the step of the signal processing unit (110) performing high-speed received data processing is described in detail.

[0156] Received signal digital conversion step

[0157] The SAR image restoration method according to GPU signal processing of the signal processing device (100) according to the present invention may include a SAR mission start step (S100). Specifically, in the SAR mission start step (S100), the SAR system, i.e., the signal processing device (100), may be initialized, and the data collection device of the RF transceiver (111) and the navigation sensor unit (112) may be activated.

[0158] The SAR image restoration method according to GPU signal processing of the signal processing device (100) according to the present invention may include a SAR received signal ADC conversion step (S200). Specifically, in the SAR received signal ADC conversion step (S200), the signal processing unit (110) may convert the received signal from the signal processing device (100) into a digital signal through an ADC (Analog-to-Digital Converter). Specifically, the RF transceiver (111) may transmit electromagnetic waves by modulating the frequency, and the RF transceiver (111) may receive the signal reflected from the target. In addition, the RF transceiver (111) may transmit the received analog signal as is to the signal processing unit (110). Accordingly, the signal received by the signal processing unit (110) may include the location and speed of the signal processing device (100) and the location and speed information of the target according to the frequency band.

[0159] According to one embodiment of the present invention, a signal processing unit (110) can control signal processing of a signal processing device (100). That is, the signal processing unit (110) can control an ADC conversion process to digitize a received analog signal. The signal processing unit (110) may include an ADC module (113), and the ADC module (113) can perform the task of converting an analog signal into a digital signal. That is, the ADC module (113) can sample the analog signal and digitize each sample to convert it into discrete digital data.

[0160] According to one embodiment of the present invention, the ADC module (113) can convert an analog signal received from the RF transceiver (111) into digital data. Specifically, the analog signal is input to the RF transceiver (111) in a continuous form, but the ADC module (113) can sample the analog signal at regular intervals to measure each instantaneous signal value. In addition, the ADC module (113) can perform quantization by converting the sampled values ​​into a digital code to represent them as discrete digital values. As a result, the continuous analog signal can be converted into digital data that can be processed by the signal processing unit (110).

[0161] According to one embodiment of the present invention, the navigation sensor unit (112) can measure the position, attitude, speed, etc. of a platform (satellite or drone) and provide navigation data necessary for SAR image restoration. That is, the signal processing unit (110) can collect the digitized SAR signal and navigation data together after DC conversion.

[0162] The SAR image restoration method according to GPU signal processing of the signal processing device (100) according to the present invention may include a pulse-by-pulse digital reception signal and navigation signal DDR memory step (S300). Specifically, in the pulse-by-pulse digital reception signal and navigation signal DDR memory step (S300), the signal processing unit (110) may synchronize the navigation data collected simultaneously with the pulse-by-pulse digital SAR signal and store it in the memory unit (120). That is, the memory unit (120) stores the received pulse-by-pulse digital SAR signal and navigation data together, so that when the mission is completed, the entire data is moved from the DDR memory of the memory unit (120) to a permanent storage device such as an SSD storage device of the data storage unit (130).

[0163] The SAR image restoration method based on GPU signal processing of the signal processing device (100) according to the present invention may include a SAR mission termination step (S400). Specifically, the SAR mission termination step (S400) may be a step in which the SAR system, i.e., the signal processing device (100), completes signal collection and the mission is terminated. When the SAR mission is terminated, all data stored in the memory unit (120) may be prepared for a subsequent processing step, and after the SAR mission termination step (S400), no new signal collection occurs, and signal processing and image restoration may proceed based on the collected data.

[0164] The SAR image restoration method according to GPU signal processing of the signal processing device (100) according to the present invention may include a step (S500) of storing digital received signals and navigation signals in an SSD for the entire mission section. Specifically, in the step (S500) of storing digital received signals and navigation signals in an SSD for the entire mission section, after the SAR mission of the signal processing device (100) is completed, the collected SAR raw data may be permanently stored in the data storage unit (130). That is, the SAR raw data may refer to digital SAR signals and navigation data for each pulse. In addition, the data storage unit (130) may include an SSD storage device, and the SSD is a device that stores SAR raw data for a long period and can store data necessary for subsequent signal processing steps so that the GPU can access the data efficiently. Accordingly, the data storage unit (130) can preserve the SAR raw data, and the processor unit (140) can utilize the preserved SAR raw data for signal processing and image restoration according to the present invention.

[0165] The SAR image restoration method according to GPU signal processing of the signal processing device (100) according to the present invention may include a GPU signal processing step (S600). Specifically, in the GPU signal processing step (S600), raw SAR data stored in the data storage unit (130) may be loaded into the processor unit (140), i.e., the GPU, and the signal processing process may begin. The processor unit (140) may perform distance compression, which is the first signal processing step of compressing the raw SAR data in the distance direction. The distance compression step may convert the raw data into the frequency domain to enhance the signal according to the distance between the target and the SAR system. Specifically, the processor unit (140) may convert the signal into the frequency domain in the distance direction through a Fourier transform (FFT), and then apply a compression filter to the raw SAR data to increase the resolution of the signal received at a specific distance.

[0166] According to one embodiment of the present invention, in the GPU signal processing step (S600), the processor unit (140) may perform an azimuth compression step to process the azimuth direction signal after distance compression. Specifically, the processor unit (140) may apply a Fourier Transform (FFT) of the azimuth direction to the collected SAR raw data to analyze the signal in the frequency domain, and perform Range Cell Migration Correction (RCMC) to correct cell migration phenomena caused by the movement of the platform. Through this correction process, signals reflected from the same ground point can be aligned to the same distance cell, and the azimuth direction resolution can be improved by applying an azimuth compression filter thereafter.

[0167] According to one embodiment of the present invention, the GPU signal processing step (S600) may include phase correction and motion correction. The processor unit (140) can correct residual phase errors and distortions caused by motion in the raw SAR data by utilizing position, attitude, and velocity data of the platform provided from the navigation sensor unit (112). Specifically, the processor unit (140) can prevent image distortion by aligning each pulse data based on a phase reference point and eliminating phase mismatches caused by minute movements of the platform.

[0168] According to one embodiment of the present invention, the GPU signal processing step (S600) can be performed by optimizing the parallel computation structure of the GPU. For example, the processor unit (140) can improve the computation speed by performing computations on a pixel-by-pixel or block-by-block basis using a CUDA-based parallel kernel. In addition, the utilization of internal GPU computation resources can be maximized by utilizing shared memory and asynchronous data transfer techniques. Through this optimization process, the present invention enables near-real-time SAR image restoration even in a limited power and hardware environment.

[0169] The SAR image restoration method according to GPU signal processing of the signal processing device (100) according to the present invention may include a SAR image restoration step (S700). Specifically, in the SAR image restoration step (S700), the signal intensity and phase are restored along with the position information of the SAR signal to generate a high-resolution surface image. The processor unit (140) can increase the resolution in the azimuth direction by performing directional compression based on data collected while the signal processing device (100) moves. Specifically, the processor unit (140) can improve the resolution of the signal received according to the movement of the signal processing device (100) through azimuth direction compression using the synthetic aperture effect. In addition, the processor unit (140) can obtain high-resolution image information by extracting detailed signals in the azimuth direction while restoring distance-compressed data into the time domain mainly through the inverse Fourier transform (iFFT).

[0170] According to one embodiment of the present invention, raw SAR data may contain background noise influenced by the surrounding environment. Therefore, a step of removing noise from the raw SAR data and correcting signal distortion according to the position and speed of the signal processing device (100) is required. The processor unit (140) may improve the quality of the final image by calculating the signal-to-noise ratio (SNR) for each pixel and filtering or correcting unwanted noise signals.

[0171] According to one embodiment of the present invention, the processor unit (140) can perform phase correction of the signal processing device (100) so that all signals are aligned at the same phase reference point. That is, the processor unit (140) can increase the accuracy of the raw SAR data through phase correction, thereby allowing the final SAR image to be restored without distortion. Accordingly, the restored SAR image shows detailed elevation information of the terrain, the size and location of the target, etc., and through this, the processor unit (140) can perform analysis and monitoring. After the final image is transmitted from the processor unit (140) to the memory unit (120), it can be stored again in the data storage unit (130) or provided to the user as needed.

[0172] According to one embodiment of the present invention, the processor unit (140) can estimate the Doppler center frequency using speed and attitude information obtained from the navigation sensor unit (112) and perform Range Cell Migration Correction (RCMC) based on the estimated value. Specifically, the processor unit (140) can correct cell migration in the time-frequency domain so that signal energy corresponding to the same ground position is aligned to the same distance cell, and accordingly, the energy focusing effect can be improved during azimuth compression.

[0173] According to one embodiment of the present invention, the processor unit (140) can perform phase correction including an autofocus technique. For example, the processor unit (140) can reduce residual phase error caused by fine maneuvers of the platform and sensor drift by applying a Phase Gradient Autofocus (PGA) or a similar technique that estimates phase error in the image domain or frequency domain. Accordingly, the contour of the detailed structure can be improved and spatial resolution maintained.

[0174] According to one embodiment of the present invention, the processor unit (140) can suppress side lobes by applying a window function during the orientation compression process and reduce spot noise by selectively applying a multi-look or speckle suppression filter to the restored image. Accordingly, the image quality of homogeneous regions is improved, and subsequent classification and detection performance can be improved.

[0175] According to one embodiment of the present invention, the processor unit (140) may be configured to perform batch-based FFT / inverse FFT, shared memory tiling, stream pipelining, and DMA-based data transfer overlap on the GPU. Specifically, block-unit loading from the data storage unit (130), kernel execution, and result storage may be overlapped as asynchronous streams to minimize latency, and mixed-precision operations may be applied as needed to reduce power consumption while increasing throughput.

[0176] According to one embodiment of the present invention, the memory unit (120) may be configured to store digital SAR signals and navigation data of each pulse by mapping them to a common timestamp. The processor unit (140) may interpolate pulse-specific navigation values ​​(position, velocity, attitude) based on the common timestamp and perform trajectory reconstruction and calculation of correction parameters based thereon. Such synchronization can increase the precision of continuous motion correction and phase correction.

[0177] According to one embodiment of the present invention, the processor unit (140) can perform radiation correction and geometric correction (geographic coordinate system alignment) on the restored SAR image, and, if necessary, apply antenna pattern correction to compensate for changes in azimuth gain. The processor unit (140) can calculate a quality index value together with the correction results and provide it as subsequent verification or transmission metadata.

[0178] FIG. 6 is a diagram illustrating the configuration of a satellite image 3D conversion system according to an embodiment of the present invention.

[0179] A satellite image 3D conversion device according to one embodiment of the present invention may include a processor unit and a memory unit, and the memory unit may store a plurality of tools for performing 3D conversion of a satellite image (such as a digital elevation model generation tool, a coordinate transformation and scaling tool, a point cloud generation tool, etc.). The processor unit may perform functions related to the 3D conversion of a satellite image by reading and executing each tool stored in the memory unit. For convenience of understanding, in the following description, the satellite image 3D conversion device is referred to as a satellite image 3D conversion system, and the process in which the processor unit of the device executes the tools stored in the memory unit to perform functions is described as "each module performs a function." This method of description is merely for convenience of explanation, and the technical scope of the present invention is not limited to the module itself.

[0180] The satellite image 3D conversion device of the present invention may be implemented by a processor unit and a memory unit within a single device, but the scope of the present invention is not limited thereto. In some embodiments, the satellite image 3D conversion system may be composed of a plurality of devices, each device including one or more of the modules of the present invention and interconnected through a network to perform the entire satellite image 3D conversion function. Accordingly, the description “a module performs a function” described below should be interpreted to include not only cases where a processor unit within a single device executes a tool stored in a memory unit, but also cases where modules distributed across a plurality of devices cooperate with each other to perform a function.

[0181] As described above, the satellite image 3D conversion system (200) of the present invention may include a digital elevation model generation module (210). Specifically, the digital elevation model generation module (210) can generate a digital surface model (DSM) including elevation data of terrain and structures based on images of satellite images. Specifically, the digital elevation model generation module (210) can align input satellite images to the same coordinate system, correct lighting and color differences, calculate pixel disparity based on aligned image pairs, and extract depth information by matching pixels at the same point by applying epipolar geometry.

[0182] At this time, the digital elevation model generation module (210) can convert the extracted depth information into an elevation value by utilizing the focal length (f) of the camera and the distance (baseline) between the two cameras, and can correct noise and outliers that may occur during this process. Accordingly, the finally generated 2D depth map represents a digital elevation model (DSM) that indicates the elevation of the terrain surface and structures. The digital elevation model generation module (210) according to the present invention can provide elevation data in various environments such as mountainous terrain, urban environments, and disaster areas.

[0183] As described above, the satellite image 3D conversion system (200) of the present invention may include a coordinate conversion and scaling module (220). Specifically, the coordinate conversion and scaling module (220) can adjust the image of the satellite image to be suitable for 3D modeling. Specifically, the coordinate conversion and scaling module (220) can convert the coordinate system based on Digital Elevation Model (DSM) data and adjust the resolution and size. At this time, since the satellite image data can use various coordinate systems, the coordinate conversion and scaling module (220) can convert it into a unified reference coordinate system and adjust it into a form suitable for creating a 3D model. In addition, the coordinate conversion and scaling module (220) can correct projection distortion that may occur in the satellite image to provide data that matches the actual terrain.

[0184] According to one embodiment of the present invention, the coordinate transformation and scaling module (220) can adjust the resolution of DSM data when it is non-uniform or the data size differs from the actual size, and can perform correction by learning the non-linear effect of elevation (Z value) on X and Y coordinate transformations. In particular, when the 3D structure is complex, such as mountainous terrain or complex buildings, the coordinate transformation and scaling module (220) can improve the precision of the data by correcting such non-linearity. In addition, the coordinate transformation and scaling module (220) can learn the complex non-linear relationship between the resolution of DSM data and projection transformations by utilizing a deep learning-based MLP (Multi-layer Perceptron) model, and can effectively correct non-uniform resolution or projection distortion where the spacing between pixels is not constant.

[0185] As described above, the satellite image 3D conversion system (200) of the present invention may include a point cloud generation module (230). Specifically, the point cloud generation module (230) may perform the role of forming a point cloud by generating points in a three-dimensional space based on a digital elevation model (DSM). The point cloud generation module (230) may generate a point cloud that represents structural information in a 3D space, such as terrain, buildings, and roads, as a set of points. Additionally, the point cloud generation module (230) may calculate the X, Y, and Z coordinates of each pixel using DSM data and calculate location information in a three-dimensional space based on data corrected in a coordinate transformation and scaling module. Furthermore, the point cloud generation module (230) may remove noise and outliers or extract key feature points such as building corners and terrain boundaries.

[0186] As described above, the satellite image 3D conversion system (200) of the present invention may include a normal vector calculation module (240). Specifically, the normal vector calculation module (240) may perform the role of defining the surface orientation of a 3D model by calculating the normal vector of each point in point cloud data. At this time, the normal vector is a vector indicating the direction in which the surface around a specific point is tilted, and can represent structural details and orientation of the 3D model. Specifically, the normal vector calculation module (240) may calculate the orientation of the surface by analyzing the relationship between a reference point and surrounding neighbor points to generate a covariance matrix, and then performing eigenvalue decomposition on it. Here, the eigenvector corresponding to the smallest eigenvalue of the covariance matrix represents the normal vector of the corresponding point, and this can define a direction perpendicular to the corresponding surface.

[0187] According to one embodiment of the present invention, the normal vector calculation module (240) can identify neighbor points around a reference point using KNN (K-Nearest Neighbors) or a radius-based neighbor algorithm, and calculate a normal vector based thereon. At this time, the calculated normal vector can be used to define the slope and directionality of the surface. For example, in the lighting and material setting step (S670), the 3D mesh generation module (260) can enhance the visual realism of the model by calculating the light reflection direction and shadow effects based on the normal vector. In addition, the normal vector calculation module (240) can express detailed directionality through the normal vector even in complex 3D structures such as mountainous terrain or building exterior walls.

[0188] As described above, the satellite image 3D conversion system (200) of the present invention may include a noise and outlier removal module (250). Specifically, the noise and outlier removal module (250) can detect and remove noise and outliers that may occur in point cloud data and a digital elevation model (DSM). Specifically, the noise and outlier removal module (250) can remove noise that is randomly generated due to sensor malfunction, light reflection, data collection errors, etc., during the satellite data or point cloud generation process, and outlier data that appears as abnormally high or low values.

[0189] According to one embodiment of the present invention, the noise and outlier removal module (250) applies an adaptive filtering technique, such as radius-based filtering or KNN-based filtering, to detect noise and outliers within the data, and can remove the detected data or perform replacement and correction operations based on the relationship with surrounding data. Through such a process, the noise and outlier removal module (250) can perform correction without degrading the quality of the data by preserving useful data as much as possible while removing noise and outliers.

[0190] As described above, the satellite image 3D conversion system (200) of the present invention may include a 3D mesh generation module (260). Specifically, the 3D mesh generation module (260) can perform the role of constructing the surface of a 3D model by generating a triangle mesh based on point cloud data. Specifically, the 3D mesh generation module (260) forms a surface by connecting each point of the point cloud in triangle units, and the generated mesh can be used as basic data to precisely represent various structures such as terrain, buildings, and roads. In addition, the 3D mesh generation module (260) can maintain surface consistency and improve data precision by compensating for gaps in the point cloud data. For example, the 3D mesh generation module (260) can efficiently process data by generating a high-density mesh in areas requiring detailed representation, such as terrain or building exterior walls, and generating a low-density mesh in simple areas.

[0191] As described above, the satellite image 3D conversion system (200) of the present invention may include a visualization module (270). Specifically, the visualization module (270) can perform the role of providing intuitive and realistic visualization by combining textures, lighting, materials, etc. based on 3D model data. Specifically, the visualization module (270) can express the color, texture, and pattern of the surface by applying texture data to the generated 3D mesh model, and can simulate optical effects such as light reflection, shadows, transparency, and refraction by reflecting lighting and material information. Through the visualization module, the user can explore the 3D model from various angles by performing operations such as rotation, zooming in / out, and movement, and natural and smooth real-time rendering is possible even in dynamic environments. In addition, the visualization module (270) can represent data in a layered manner, allowing specific data such as the exterior walls of a building, internal structure, and terrain to be selectively highlighted or hidden.

[0192] According to one embodiment of the present invention, the visualization module (270) can realistically represent the interaction of light on the surface of objects of various materials, such as metal, glass, and concrete, by utilizing lighting and material data generated from a model such as NeRF (Neural Radiance Fields). In addition, the visualization module (270) can intuitively represent data by reflecting intensity and color information obtained from SAR (radar image) and optical image, and can provide precise visualization through a high-resolution output function.

[0193] FIG. 7 is a diagram illustrating the steps for performing a satellite image 3D conversion system according to an embodiment of the present invention. FIG. 8 is a diagram illustrating the steps for generating a digital elevation model according to an embodiment of the present invention.

[0194] According to the present invention, a Digital Surface Model (DSM) may refer to 3D data including X and Y coordinates and an altitude (Z coordinate) of each point in a satellite image. Accordingly, a DSM can be generated in the step where the Z coordinate is calculated and this value is combined with the X and Y coordinates. Specifically, in the following Z coordinate calculation step (S26), the digital surface model generation module (210) can calculate the actual Z coordinate (altitude) of each pixel by utilizing camera parameters based on the depth (Disparity) data obtained in the stereo matching step (S24). Accordingly, the digital surface model generation module (210) can generate a digital surface model (DSM) by combining the calculated Z coordinate with the X and Y coordinates.

[0195] Satellite image data loading step (S21)

[0196] According to one embodiment of the present invention, the digital elevation model generation module (210) can correct the brightness or contrast of the satellite image by utilizing a CNN model (S21-1) to correct quality issues (distortion, noise, brightness / contrast imbalance, etc.) of the satellite image and to increase the accuracy of digital elevation model generation. Additionally, the digital elevation model generation module (210) can remove blur and noise from the satellite image and recover distorted pixel data. Furthermore, if there are missing pixels or damaged areas in the satellite image data, the digital elevation model generation module (210) can predict pixel values ​​by learning information around the damaged area using a CNN model (S21-1). That is, the digital elevation model generation module (210) can perform the role of improving the quality of the satellite image during the satellite image data loading step (S21).

[0197] Satellite image alignment step (S22)

[0198] According to one embodiment of the present invention, the digital elevation model generation module (210) can remove geometric distortion of a satellite image and perform conversion to a geographic coordinate system. That is, the digital elevation model generation module (210) can correct distorted images by utilizing satellite sensor data and GPS information, and also correct distortion caused by terrain relief to generate an image with accurate location information. Subsequently, the digital elevation model generation module (210) can perform the task of aligning multiple images of the same area taken from various viewpoints. Specifically, the digital elevation model generation module (210) can extract and match feature points between images and analyze the relationships between feature points using algorithms such as SIFT and ORB. In addition, the digital elevation model generation module (210) can calculate a transformation matrix (Homography Matrix) based on the analysis results and correct rotation, translation, and scaling between images to perform alignment into a single reference coordinate system.

[0199] According to one embodiment of the present invention, the digital elevation model generation module (210) may perform normalization on image resolution and shooting conditions to increase the consistency of aligned data. That is, the digital elevation model generation module (210) may convert data of different resolutions to the same resolution and correct differences in brightness and color according to the shooting environment.

[0200] According to one embodiment of the present invention, a digital elevation model generation module (210) can align satellite images by utilizing an STN (Spatial Transformer Networks) model (S22-1) and a Homography Networks model (S22-2). Specifically, the STN model (S22-1) can learn the transformation process from the input satellite image to automatically correct distortion and perform the role of aligning the image to the same reference coordinate system. That is, the STN model (S22-1) can learn and correct geometric distortions (rotation, translation, scale change) included in the satellite image, and can transform the image generated depending on the shooting angle or position of the satellite sensor into a normalized state. In addition, the STN model (S22-1) can learn the features of the satellite image and perform a projection transformation. For example, the STN model (S22-1) can perform a transformation of an image captured by satellite sensor data (UAV, drone, satellite, etc.) into a reference coordinate system of the actual terrain. In addition, the STN model (S22-1) can align local or regional features such as buildings, roads, and mountainous terrain in satellite images.

[0201] According to one embodiment of the present invention, a digital elevation model generation module (210) can align satellite images by utilizing a Homography Networks model (S22-2). Specifically, the Homography Networks model (S22-2) learns the relationships between satellite images to automate feature point-based geometric alignment and perform projection transformation, i.e., Homography Matrix calculation. That is, the Homography Networks model (S22-2) can extract feature points from satellite images, match identical points between images, and learn the geometric relationships between two images to calculate a Homography Matrix (projection transformation matrix). Such a matrix can represent transformation information including rotation, translation, scaling, and distortion correction between images. In addition, the Homography Networks model (S22-2) can learn multiple images of the same region taken from various viewpoints to perform alignment into a single reference coordinate system. Accordingly, the present invention can eliminate discrepancies between satellite images and provide data suitable for generating a digital elevation model.

[0202] Throughout this specification, any part stating that each module of the satellite image 3D conversion device performs a specific function should be interpreted not to mean that the module performs the function independently, but to mean that the processor unit of the satellite image 3D conversion device of the present invention performs the function by executing the corresponding module (program, model, algorithm, etc.) stored in the memory unit. That is, in the following description, all parts stating that “the module performs a function” refer to the processor unit of the device realizing the function by calling and executing the corresponding module stored in the memory unit.

[0203] Satellite image matching step (S23)

[0204] According to one embodiment of the present invention, the digital elevation model generation module (210) can perform the role of extracting feature points in the satellite image matching step (S23). Specifically, the digital elevation model generation module (210) can detect local feature points to be used for image matching by extracting unique and distinguishable feature points from each satellite image. For example, the digital elevation model generation module (210) can extract building corners or road intersections, and can use the SIFT (Scale-Invariant Feature Transform) algorithm or the ORB (Oriented FAST and Rotated BRIEF) algorithm.

[0205] According to one embodiment of the present invention, the digital elevation model generation module (210) can perform the role of feature point matching in the satellite image matching step (S23). Specifically, the digital elevation model generation module (210) can match identical points between different images and can generate descriptors for each feature point to represent them as unique vectors. In addition, the digital elevation model generation module (210) can perform matching by calculating the similarity between feature point vectors between satellite images and can utilize the Nearest Neighbor Search technique. Furthermore, the digital elevation model generation module (210) can remove outlines to improve precision by removing outliers among the matched points.

[0206] According to one embodiment of the present invention, the digital elevation model generation module (210) can learn and calculate the geometric relationship between two images based on the feature points matched in the satellite image matching step (S23). Specifically, the digital elevation model generation module (210) can express projection transformation relationships including translation, rotation, and scaling between images, and in this case, can perform Homography Matrix calculation after removing outliers through RANSAC (Random Sample Consensus). In addition, the digital elevation model generation module (210) can perform alignment by calculating an Affine Matrix when only local transformation is required, and can also perform transformation to the same coordinate system by warping the image by applying the calculated transformation matrix.

[0207] According to one embodiment of the present invention, the digital elevation model generation module (210) can correct pixel values ​​in the overlapping area to integrate the overlapping images after geometric transformation and generate a single continuous data. Additionally, the digital elevation model generation module (210) can correct differences in brightness and contrast between multiple images to generate a uniform result. Accordingly, the digital elevation model generation module (210) can complete the combined area by interpolating surrounding pixel data for the empty area remaining after alignment.

[0208] According to one embodiment of the present invention, a digital elevation model generation module (210) can generate unique feature vectors from image fragments of two satellite images by utilizing a Siamese Network model (S23-1). Specifically, the Siamese Network model (S23-1) can perform mapping from two images to the same feature vector space through the network's twin branch structure. Additionally, the Siamese Network model (S23-1) can match identical points (feature points) by calculating the similarity between pairs of patches in each image. Furthermore, the Siamese Network model (S23-1) can align multiple images by finding common feature points in multiple images.

[0209] According to one embodiment of the present invention, a digital elevation model generation module (210) can identify potential matching points by calculating the similarity between feature vectors extracted from two satellite images using a Deep feature matching model (S23-2). Additionally, the Deep feature matching model (S23-2) can extract features and perform matching at various scales by considering the resolution difference between images, and can perform matching even on images of multiple resolutions.

[0210] Stereo matching step (S24)

[0211] According to one embodiment of the present invention, the digital elevation model generation module (210) can align satellite images input for stereo matching into the same coordinate system. Specifically, the digital elevation model generation module (210) can align two satellite images based on the same reference coordinate system and can improve matching accuracy by correcting for differences in lighting and color between the two satellite images. Subsequently, the digital elevation model generation module (210) can extract depth information by calculating the pixel displacement (Disparity) of the same point between the two satellite images.

[0212] In this case, the digital elevation model generation module (210) can apply epipolar geometry to set the same point to be located on the epipolar line based on the shooting position and angle of the two satellite images, and can calculate the difference in pixel positions of the same object in the two satellite images. Additionally, the digital elevation model generation module (210) can match the pixel with the highest similarity within the search area. Subsequently, the digital elevation model generation module (210) can calculate the similarity cost of each pixel and extract the optimal parallax value based on the cost volume. Accordingly, the digital elevation model generation module (210) can estimate depth (altitude) information based on the calculated parallax value and calculate the depth value using the camera's focal length (f) and the distance between the two cameras (baseline). That is, the digital elevation model generation module (210) can generate a 2D depth map by converting the parallax value of each pixel into a depth value.

[0213] According to one embodiment of the present invention, features of left (L) and right (R) stereo images can be extracted using a GCNet model (Geometry and Context Network) (S24-1), and the matching cost of each pixel pair can be calculated for all possible disparity values. Additionally, the GCNet model (S24-1) can convert the matching cost into a 3D Cost Volume and learn the features of the cost volume using 3D Convolutional Neural Networks (3D CNN). Subsequently, the GCNet model (S24-1) can extract the optimal disparity of each pixel from the cost volume.

[0214] According to one embodiment of the present invention, a cost volume can be generated based on the multiscale features of the left (L) and right (R) images by utilizing a PSMNet model (Pyramid Stereo Matching Network) (S24-2). At this time, the cost volume generated by the PSMNet model (S24-2) can be used as input data for depth (disparity) estimation. Additionally, the PSMNet model (S24-2) can refine the cost volume using a 3D CNN and calculate disparity from the refined cost volume to accurately predict the disparity value of each pixel. Furthermore, the PSMNet model (S24-2) can generate a disparity map by extracting the optimal disparity value from the cost volume. In this case, the generated disparity map contains three-dimensional structural information of the stereo image and can be used to generate a digital elevation model (DSM).

[0215] Depth information verification and noise removal step (S25)

[0216] According to one embodiment of the present invention, the digital elevation model generation module (210) can perform a depth information verification and noise removal step (S25). Specifically, the digital elevation model generation module (210) can increase the reliability of the data by detecting and removing noise and outliers of depth information for the depth map generated in the stereo matching step (S24). For example, the digital elevation model generation module (210) can consider data that falls outside the statistical range of depth values ​​as outliers and can remove sudden depth changes or values ​​that are inconsistent with surrounding pixels. In addition, the digital elevation model generation module (210) can detect noise by identifying abnormal data based on surrounding spatial relationships.

[0217] According to one embodiment of the present invention, the digital elevation model generation module (210) can perform data refinement to generate a smooth and consistent map by refining depth data after noise removal. That is, the digital elevation model generation module (210) can perform refinement so that depth values ​​between adjacent pixels naturally connect, and can perform interpolation to have a locally consistent surface. In addition, the digital elevation model generation module (210) can perform interpolation using surrounding pixel data for areas where depth information is missing. For example, the digital elevation model generation module (210) can replace the nearest valid pixel value for an area where depth information is missing, or apply the average of surrounding values.

[0218] According to one embodiment of the present invention, a digital elevation model generation module (210) can perform noise detection and noise removal by utilizing an Autoencoder model (S25-1). Specifically, the Autoencoder model (S25-1) can learn normal depth data patterns, and unlearned outliers and noise may exhibit high reconstruction errors during the reconstruction process. Additionally, the Autoencoder model (S25-1) can remove noise from input data and preserve information from the original data. Furthermore, the Autoencoder model (S25-1) can preserve the structure of the data through a learned low-dimensional latent space and ignore noise elements of the input. For example, if a depth map containing noise is input, the Autoencoder model (S25-1) can output a reconstructed clean depth map. Additionally, the Autoencoder model (S25-1) can detect outliers based on reconstruction errors and perform removal. Specifically, the Autoencoder model (S25-1) considers pixels whose reconstruction error exceeds a specific threshold as outliers and can replace these values ​​with surrounding data or perform interpolation.

[0219] According to one embodiment of the present invention, a digital elevation model generation module (210) can perform noise removal and depth information generation by utilizing a GAN model (S25-2). Specifically, the GAN model (S25-2) can learn depth information to generate a refined depth map with noise removed, and can perform restoration based on learned patterns for areas with lost data or outliers. Additionally, the GAN model (S25-2) can compensate for pixels with abnormally large or small values ​​according to surrounding data patterns.

[0220] Z-coordinate calculation step (S26)

[0221] According to one embodiment of the present invention, a digital elevation model generation module (210) can calculate the Z-coordinate (altitude) of each pixel in 3D space for a depth map generated in a stereo matching step (S24) or a depth information verification and noise removal step (S25). In this case, the digital elevation model generation module (210) can calculate the Z-coordinate using a Multi-Layer Perceptron (MLP) model (S26-1). Specifically, the MLP model (S26-1) can calculate the Z-coordinate by learning a non-linear relationship based on input data. Here, the MLP model (S26-1) utilized by the digital elevation model generation module (210) can be stored in the memory unit of the satellite image 3D conversion device of the present invention, and the processor unit of the device can perform the alignment function by executing it. Therefore, the part of this specification stating that “the module performs alignment using an MLP model” means that the processor unit of the device calls and executes the MLP model stored in the memory unit to perform the corresponding function.

[0222] According to one embodiment of the present invention, an MLP model (S26-1) can calculate a Z-coordinate using a depth map and camera parameters (focal length f, distance between two cameras B) as input data. Specifically, the depth map includes information on the amount of movement between pixels calculated in the stereo matching step (S24) and can be converted into a range suitable for the model through normalization. Here, the input of the MLP model (S26-1) may consist of a parallax value (d), a focal length (f), and a baseline distance (B). Specifically, the MLP model (S26-1) may consist of an input layer, a hidden layer, and an output layer, and an activation function such as ReLU or Leaky ReLU may be used in the hidden layer. Additionally, the MLP model (S26-1) learns a non-linear relationship between the parallax and camera parameters in the hidden layer and uses a linear activation function in the output layer to finally predict the depth value (Z) of each pixel.

[0223] According to one embodiment of the present invention, the MLP model (S26-1) may use mean squared error (MSE) or mean absolute error (MAE) as the loss function during the learning process. Accordingly, the MLP model (S26-1) can minimize the difference between the predicted Z-coordinate and the actual depth value.

[0224] FIG. 9 is a diagram illustrating coordinate transformation and scaling steps according to an embodiment of the present invention.

[0225] Pixel coordinate extraction step (S31)

[0226] According to one embodiment of the present invention, the coordinate transformation and scaling module (220) can perform a transformation into X, Y coordinates on a 2D plane based on row and column information of each pixel in the DSM data to extract each pixel location (X, Y) of the DSM data and the value stored in the corresponding pixel. In addition, the coordinate transformation and scaling module (220) can extract altitude information (Z value) or pixel brightness value of each pixel.

[0227] According to one embodiment of the present invention, the coordinate transformation and scaling module (220) can perform a transformation of the pixel coordinates (X, Y) of the DSM data into a geographic coordinate system in order to map the pixel coordinates (X, Y) of the DSM data to an actual geographic coordinate system. For example, the coordinate transformation and scaling module (220) can perform a mapping of the rows and columns of the DSM to latitude and longitude values. In addition, the coordinate transformation and scaling module (220) can calculate the actual distance of each pixel by reflecting the pixel size (Spatial Resolution) of the DSM data. Subsequently, the coordinate transformation and scaling module (220) can normalize the pixel coordinate data and normalize the X, Y coordinates to transform them into a certain range. For example, the coordinate transformation and scaling module (220) can normalize the X, Y coordinates to transform them into [-1, 1].

[0228] Z coordinate extraction step (S32)

[0229] According to the present invention, the coordinate transformation and scaling module (220) can perform a Z-coordinate extraction step (S32). Specifically, the coordinate transformation and scaling module (220) can extract elevation information for each pixel from the DSM data. Here, since the coordinate transformation and scaling module (220) extracts a Z-value (elevation value) that has already been calculated from the DSM data, it should be distinguished from the digital elevation model generation module (210) in the Z-coordinate calculation step (S26) newly calculating the Z-value using the depth map generated in the stereo matching step (S24).

[0230] According to one embodiment of the present invention, the coordinate transformation and scaling module (220) can extract the altitude (Z value) stored in each pixel from the DSM data. Additionally, the coordinate transformation and scaling module (220) can standardize the range of the Z value and perform transformations such as normalization, unit conversion, and scaling accordingly. Specifically, the coordinate transformation and scaling module (220) can convert the Z value to a specific range and perform normalization to a range such as [0, 1]. Additionally, if the altitude value is stored in centimeters, the coordinate transformation and scaling module (220) can convert it to meters and can perform transformations in a form suitable for analysis and visualization by adjusting the scale of the altitude value. Additionally, the coordinate transformation and scaling module (220) can combine the extracted Z value with X and Y pixel coordinates to generate data to be used for 3D coordinate transformation. For example, the coordinate transformation and scaling module (220) can combine the Z value of each pixel with the X and Y coordinates to form a three-dimensional array, and can perform alignment so that each Z value matches the X and Y coordinates of the corresponding pixel.

[0231] According to one embodiment of the present invention, the coordinate transformation and scaling module (220) can also perform the role of identifying and removing incorrect Z values ​​(outliers, noise) in DSM data. Specifically, the coordinate transformation and scaling module (220) can detect when an altitude value has an abnormally high or low value. For example, if a value excessively large compared to the building altitude is found in the DSM data, or if a negative value is found in the DSM data, the coordinate transformation and scaling module (220) can identify and remove it as an incorrect Z value. In addition, the coordinate transformation and scaling module (220) can remove noise by applying a Gaussian Filter or a Median Filter to the DSM data, and can interpolate missing values ​​(NaN) with surrounding data or replace them with default values.

[0232] 3D coordinate transformation step (S33)

[0233] According to one embodiment of the present invention, a coordinate transformation and scaling module (220) can construct 3D data by integrating X, Y coordinates and a Z value (altitude). Specifically, the coordinate transformation and scaling module (220) can generate a 3D coordinate array in the form of [X, Y, Z] by taking X, Y coordinates (pixel location) and a Z value (altitude information) as inputs. Accordingly, the coordinate transformation and scaling module (220) can convert X, Y pixel coordinates and Z values ​​into actual spatial coordinates. Specifically, considering the resolution of the DSM data, X, Y coordinates can be converted into actual distances, and if the Z value (altitude) is stored as centimeters, pixel values, etc., it can be converted into actual units such as meters.

[0234] According to one embodiment of the present invention, the coordinate transformation and scaling module (220) can perform mapping of the transformed 3D coordinates to an actual geographical coordinate system. Specifically, the coordinate transformation and scaling module (220) can perform transformations to fit the geographical coordinate system using existing X, Y, and Z data. For example, the coordinate transformation and scaling module (220) can perform transformations into latitude, longitude, and height using existing X, Y, and Z data. Additionally, the coordinate transformation and scaling module (220) can adjust the alignment and position of the entire data by setting a reference point during coordinate transformation.

[0235] According to one embodiment of the present invention, the coordinate transformation and scaling module (220) can transform 3D coordinates by utilizing a Multi-Layer Perceptron (MLP) model (S33-1). Specifically, the coordinate transformation and scaling module (220) can learn the complex non-linear relationship between the spatial resolution and projection transformation of DSM data by utilizing the MLP model (S33-1). For example, the coordinate transformation and scaling module (220) can perform correction by utilizing the MLP model (S33-1) when the spacing between pixels is not constant (non-uniform resolution) or when there is projection distortion. Additionally, the coordinate transformation and scaling module (220) can learn the non-linearity in which altitude (Z value) affects X and Y coordinate transformations according to height by utilizing the MLP model (S33-1). For example, the MLP model (S33-1) can perform corrections regarding the effect of Z-value fluctuations on coordinate transformation in complex 3D structures such as mountainous terrain or buildings. Here, the MLP model (S33-1) utilized by the coordinate transformation and scaling module (220) can be stored in the memory unit of the satellite image 3D conversion device of the present invention, and the alignment function can be performed by the processor unit of the device executing it. Accordingly, the part of the specification stating “the module performs alignment using the MLP model” means that the processor unit of the device calls and executes the MLP model stored in the memory unit to perform the corresponding function.

[0236] According to one embodiment of the present invention, the coordinate transformation and scaling module (220) can convert pixel coordinates into latitude, longitude, and altitude by utilizing an MLP model (S33-1) to non-linearly process geographic projection transformations, and can also learn the ability to automatically filter out abnormally large or small values ​​(outliers) during the learning process. Accordingly, when a missing value (NaN) or an incorrect value is provided as input, the MLP model (S33-1) can generate consistent 3D coordinates by correcting based on surrounding data. In addition, the MLP model (S33-1) can output refined 3D coordinates using pixel coordinates and Z values ​​as input data.

[0237] Z coordinate scaling step (S34)

[0238] According to one embodiment of the present invention, the coordinate transformation and scaling module (220) may limit the range of values ​​through clipping or dynamically adjust the range for specific analysis and visualization purposes when the Z-coordinate value extracted in the Z-coordinate extraction step (S32) is outside the expected range. In this case, the coordinate transformation and scaling module (220) may remove outliers such as abnormally large or small values, and may compensate for missing values ​​using interpolation with surrounding data.

[0239] According to one embodiment of the present invention, the coordinate transformation and scaling module (220) can perform scaling by utilizing a Recurrent Neural Network (RNN) model (S34-1). Specifically, since Z-coordinate data has continuous terrain characteristics and elevation values ​​between adjacent pixels may follow a constant pattern, the RNN model (S34-1) can learn spatial continuity to generate smoother and more consistent results during the scaling process of Z values. For example, the RNN model (S34-1) can consider each row or column in the DSM data as sequential data and process it to generate a scaling value that reflects continuity. Additionally, the RNN model (S34-1) can predict the scaling value of the current pixel by considering the Z value of the previous pixel and can mitigate discontinuous elevation changes (noise). Here, the RNN model (S34-1) utilized by the coordinate transformation and scaling module (220) may be stored in the memory unit of the satellite image 3D conversion device of the present invention, and the processor unit of the device may perform the alignment function by executing it. Accordingly, the part of the specification stating “the module performs alignment by utilizing the RNN model (S34-1)” means that the processor unit of the device calls and executes the RNN model (S34-1) stored in the memory unit to perform the corresponding function.

[0240] According to one embodiment of the present invention, the RNN model (S34-1) can compensate for missing values ​​(NaN) or outliers that may occur in Z values. Additionally, since the RNN model (S34-1) learns the interrelationships between consecutive pixels, it can perform compensation based on surrounding values ​​even if there are missing values. For example, the RNN model (S34-1) can compensate for missing Z coordinates by considering previous and subsequent pixel values, and can also adjust sudden changes in Z values ​​(noise) to match surrounding pixel patterns. Furthermore, the RNN model (S34-1) can perform data standardization by normalizing Z coordinates to a specific range or converting them to actual units such as meters. For example, the RNN model (S34-1) can learn the overall distribution of Z values, map each Z value to a normalized range, and maintain continuity between pixels so that the normalized data is smoothly connected.

[0241] According to one embodiment of the present invention, an RNN model (S34-1) can perform dynamic scaling by reflecting the regional characteristics of elevation values ​​in terrain data. For example, the RNN model (S34-1) can learn the characteristics of Z values ​​in various terrains, such as mountainous areas, flat land, and buildings, and dynamically apply a scaling value suitable for the corresponding area. In addition, the RNN model (S34-1) can appropriately correct the Z value of a specific area based on the context of surrounding data.

[0242] FIG. 10 is a diagram illustrating the step of generating a point cloud and calculating a normal vector according to an embodiment of the present invention.

[0243] Point cloud generation step (S41)

[0244] According to one embodiment of the present invention, a point cloud generation module (230) can generate a point in 3D space by integrating X, Y, and Z coordinate data into one. Specifically, the point cloud generation module (230) can construct 3D point data in the form of [X, Y, Z] by combining coordinate data generated in the coordinate transformation and scaling step (S30). Accordingly, the point cloud generation module (230) can generate individual points based on the 3D coordinates of each pixel and aggregate them into a point cloud. Here, each point includes X, Y, and Z coordinates, and may also include attributes such as RGB colors or intensity values.

[0245] According to one embodiment of the present invention, a point cloud generation module (230) can generate a point cloud by utilizing a PointNet model (S42-1). Specifically, the PointNet model (S42-1) can learn spatial patterns from input 3D coordinate data to refine individual points of the point cloud or extract features. That is, the PointNet model (S42-1) can be suitable for processing irregular point data and can generate unique embeddings for the location and attributes of each point.

[0246] According to one embodiment of the present invention, a point cloud generation module (230) can generate a point cloud by utilizing a Dynamic Graph Convolutional Neural Network (DGCNN) model (S42-2). Specifically, the DGCNN model (S42-2) can consider each point of the point cloud as a node of the graph and define the relationship between points as edges of the graph. Such a graph is dynamically updated and can learn the local relationships of the point cloud. Therefore, the DGCNN model (S42-2) can effectively process dense areas by considering the spatial relationships of the points and can refine noisy point data.

[0247] Neighbor point search step (S42)

[0248] According to the present invention, the point cloud generation module (230) can perform a neighbor point search step (S42) for each point of the point cloud to calculate the direction of the surface (normal vector) based on neighboring neighbor points. Specifically, the point cloud generation module (230) can calculate the characteristics of a local region containing each point through the neighbor point search. For example, the point cloud generation module (230) can calculate curvature, surface flatness, point density, etc. through the neighbor point search. In addition, the point cloud generation module (230) can define the relationship of how each point is connected to neighboring points and can learn structural relationships by generating a graph between points.

[0249] According to one embodiment of the present invention, a point cloud generation module (230) may utilize a K-Nearest Neighbors (KNN) algorithm to search for the closest points within a specific distance (radius) or based on the number of points centered on each point. Specifically, the point cloud generation module (230) may search for K closest neighbor points of each point and may use Euclidean distance or Manhattan distance criteria. Additionally, the point cloud generation module (230) may search for points located within a certain radius (radius) from each point. For example, the point cloud generation module (230) may learn density-based features by searching for all points within a radius of 5m.

[0250] According to one embodiment of the present invention, the point cloud generation module (230) can quantitatively identify local relationships by calculating the distance between each point and neighboring points. In this case, the point cloud generation module (230) may utilize a Euclidean distance calculation method and may select the top K points by sorting from the nearest point. Subsequently, the point cloud generation module (230) can structure and store the relationship between each point and neighboring points.

[0251] Normal vector calculation step (S43)

[0252] According to one embodiment of the present invention, a normal vector calculation module (240) receives point cloud data and neighbor point information (KNN or radius-based neighbors) as input and can calculate a normal vector based on the relationship between a reference point and neighbor points. Specifically, the normal vector calculation module (240) can generate a covariance matrix by centering the neighbor points of each reference point and perform eigenvalue decomposition on it to determine the orientation of the virtual surface of a 3D object extracted from the point cloud data. Here, the eigenvector corresponding to the smallest eigenvalue of the covariance matrix can represent the normal vector of the corresponding point.

[0253] According to one embodiment of the present invention, the normal vector calculation module (240) may align the direction of the normal vector so as to face outward from a reference point, and may store the aligned vector by flipping it in the opposite direction. Here, the normal vector calculation module (240) can calculate normal vectors even on more complex surfaces by utilizing a Graph Neural Network (GNN) model (S43-1) and a PointNet++ model (S43-2). Specifically, the GNN model (S43-2) can estimate normal vectors based on local patterns by learning the relationship between each point and neighboring points as a graph structure. Additionally, the PointNet++ model (S43-2) can calculate precise normal vectors even on complex surfaces with high curvature by learning the local features of the point cloud.

[0254] FIG. 11 is a diagram illustrating a noise and outlier removal step according to an embodiment of the present invention.

[0255] Noise detection step (S51)

[0256] According to one embodiment of the present invention, a noise and outlier removal module (250) can identify noise and outliers included in point cloud data. Specifically, the noise and outlier removal module (250) can define criteria for noise in the point cloud data. For example, the noise and outlier removal module (250) can identify points with abnormally low density compared to surrounding points as data density outliers and detect them as noise. Additionally, the noise and outlier removal module (250) can identify points that deviate significantly from the expected space as location outliers and detect them as noise. Furthermore, the noise and outlier removal module (250) can identify points with RGB or intensity values ​​that do not match surrounding points as attribute outliers and detect them as noise.

[0257] According to one embodiment of the present invention, the noise and outlier removal module (250) can detect abnormal points (noise) in the data and can perform detection by applying a K-Nearest Neighbors (KNN) algorithm. Specifically, the noise and outlier removal module (250) can calculate the neighbor point density of each point and perform a comparison with the surrounding density, and can consider points below a specific threshold as noise. In addition, the noise and outlier removal module (250) can analyze the density and location distribution of points using a density-based clustering (DBSCAN) technique and separate points with low density as noise. In addition, the noise and outlier removal module (250) can detect noise by comparing the average distance between surrounding points with a reference distance using a distance-based analysis technique. Furthermore, the noise and outlier removal module (250) can add tags to the detected noise points so that they can be utilized in the removal process. That is, the noise and outlier removal module (250) can assign a 'noise' tag to points identified as noise.

[0258] According to one embodiment of the present invention, the noise and outlier removal module (250) can detect noise by utilizing an Autoencoder model (S51-1). Specifically, the noise and outlier removal module (250) can identify noise by learning complex patterns of input data using an Autoencoder model (S51-1), and can detect points with large reconstruction errors as noise while restoring the main structure of the data.

[0259] According to one embodiment of the present invention, the noise and outlier removal module (250) can detect noise by utilizing a Graph Neural Network (GNN) model (S51-2). Specifically, the GNN model (S51-2) can detect noise by learning the spatial relationships between points based on a graph, and can consider the relationships between neighbors of the points and global patterns together.

[0260] According to one embodiment of the present invention, the noise and outlier removal module (250) can detect noise by utilizing a Convolutional Neural Networks (CNN) model (S51-3). Specifically, the CNN model (S51-3) can learn spatial patterns by converting point cloud data into 3D images. Additionally, the CNN model (S51-3) can detect noise based on the geometric distribution between neighboring points.

[0261] Noise removal step (S52)

[0262] According to one embodiment of the present invention, the noise and outlier removal module (250) may set a removal criterion to efficiently remove noise in the noise removal step (S52). Specifically, the noise and outlier removal module (250) may set a criterion to remove a point when the surrounding point density is lower than a specific threshold. Additionally, the noise and outlier removal module (250) may perform removal when the average distance or maximum distance to surrounding points exceeds a set threshold. Furthermore, the noise and outlier removal module (250) may remove a point whose RGB value or intensity is extremely different from surrounding points.

[0263] According to one embodiment of the present invention, the noise and outlier removal module (250) can remove detected noise data from the point cloud based on a preset criterion. Specifically, the noise and outlier removal module (250) can delete points identified as noise from the point cloud data. Additionally, the noise and outlier removal module (250) can generate a new point cloud data array that does not include noise points. Accordingly, the noise and outlier removal module (250) can utilize interpolation to compensate for data gaps (missing values) that may occur due to noise removal. Specifically, to compensate for data gaps (missing values) that may occur due to noise removal, the noise and outlier removal module (250) can compensate for missing values ​​using data around the removed noise points, and can also generate new points using average values ​​or predicted values ​​in the removed area.

[0264] Outlier detection step (S53)

[0265] According to one embodiment of the present invention, the noise and outlier removal module (250) may set criteria for identifying outliers in the outlier detection step (S53). For example, the noise and outlier removal module (250) may consider a point as an outlier if the average distance or maximum distance from surrounding points exceeds a threshold value. Additionally, the noise and outlier removal module (250) may consider a point as an outlier if the point density in a specific area is extremely lower or higher than the surrounding area. Furthermore, the noise and outlier removal module (250) may consider a point as an outlier if its attributes, such as RGB values ​​or intensity, are abnormally different from those of surrounding points. Additionally, the noise and outlier removal module (250) may consider a point as an outlier if its coordinates or attributes exceed a predetermined standard deviation from the average of the entire data.

[0266] According to one embodiment of the present invention, the noise and outlier removal module (250) can identify outliers according to set criteria. Specifically, the noise and outlier removal module (250) can detect points deviating from the distance distribution as outliers by calculating the distance of each point to its K-Nearest Neighbors. Additionally, the noise and outlier removal module (250) can classify points in areas with low point density as outliers. Furthermore, the noise and outlier removal module (250) can detect outliers by comparing the local density of each point with that of surrounding points. Additionally, the noise and outlier removal module (250) can identify points exceeding a threshold value by analyzing the mean and variance of the distance between each point and surrounding points.

[0267] Outlier handling step (S54)

[0268] According to one embodiment of the present invention, the noise and outlier removal module (250) can generate refined data by removing outlier data from the point cloud in the outlier processing step (S54). Specifically, the noise and outlier removal module (250) can simply remove distance-based outliers or density-based outliers from the point cloud array. Additionally, after removing outliers, the noise and outlier removal module (250) can maintain data continuity by interpolating the data of surrounding points.

[0269] According to one embodiment of the present invention, the noise and outlier removal module (250) may modify data by utilizing the average or pattern of surrounding data without removing outliers. Specifically, the noise and outlier removal module (250) may perform neighbor mean correction by replacing the value of a detected outlier with the average value of surrounding points. Additionally, the noise and outlier removal module (250) may perform regression model correction to predict and correct outlier values ​​by learning the relationship between outliers and surrounding points. Furthermore, the noise and outlier removal module (250) may correct outliers by utilizing an Autoencoder model or a GNN model.

[0270] FIG. 12 is a diagram illustrating the steps for generating a 3D mesh and constructing a high-resolution 3D model according to an embodiment of the present invention.

[0271] Triangle mesh generation step (S61)

[0272] According to one embodiment of the present invention, the triangle mesh generation step (S61) in the satellite image 3D conversion system (200) can be performed by a 3D mesh generation module (260). Specifically, the 3D mesh generation module (260) can generate a triangle mesh by utilizing 3D coordinate data provided by the point cloud generation module (230). Here, a triangle mesh is a data structure used in 3D computer graphics and modeling, which means representing the surface of a 3D object as triangles, and may be a method of constructing a surface by connecting vertices in 3D space in triangular units.

[0273] Surface activation step (S62)

[0274] According to one embodiment of the present invention, the surface activation step (S62) can be performed by a 3D mesh generation module (260). Specifically, the 3D mesh generation module (260) can activate the surface of the 3D mesh formed in the triangle mesh generation step (S61) to create a more detailed and consistent surface structure.

[0275] According to one embodiment of the present invention, the 3D mesh generation module (260) can improve the overall quality by smoothing the surface of the 3D mesh and correcting distortions or irregular parts. To this end, the 3D mesh generation module (260) can utilize normal vector information indicating the surface direction provided by the normal vector calculation module (240), and can adjust the density of the mesh based on the orientation of the surface or remove unnecessary meshes.

[0276] According to one embodiment of the present invention, the 3D mesh generation module (260) may utilize a surface smoothing technique to create a more natural surface by mitigating protrusions or curvatures on the mesh surface, a reconstruction technique to provide a consistent mesh structure by correcting defective triangle meshes, and a subdivision technique to form a more detailed 3D surface by increasing the resolution of triangle meshes.

[0277] Hole filling step (S63)

[0278] According to one embodiment of the present invention, the hole filling step (S63) can be performed by a 3D mesh generation module (260), and the 3D mesh generation module (260) can detect and fill defects, i.e., holes, that occur on the surface of the 3D mesh. Such holes may occur due to incompleteness of satellite image data or data loss during the mesh generation process. Specifically, the 3D mesh generation module (260) may utilize a GNN model (S63-1) and a MeshCNN model (S63-2) to detect and fill holes.

[0279] According to one embodiment of the present invention, a 3D mesh generation module (260) can learn the spatial relationship between vertices and edges around a defect by utilizing a Graph Neural Network (GNN) model (S63-1) to represent the connection relationship between triangles of a 3D mesh in a graph form. Additionally, the 3D mesh generation module (260) can restore the connectivity of the mesh based on structural features around the hole using a GNN model and fill the hole by generating new vertices and triangles needed for the defect. For example, the 3D mesh generation module (260) can fill the defect by utilizing a GNN model to learn the coordinates of vertices existing around the hole and generating a triangle mesh through this.

[0280] According to one embodiment of the present invention, the 3D mesh generation module (260) can compensate for defects by analyzing the triangle mesh pattern around the hole using a MeshCNN model. Specifically, the 3D mesh generation module (260) can predict the structure of the area containing the hole by analyzing the slope, angle, length, etc. between triangles using a MeshCNN model and generate a natural triangle mesh. In particular, the 3D mesh generation module (260) can fill in defects by strengthening the triangle connectivity around the hole and maintaining consistency of the mesh surface using a MeshCNN model.

[0281] Mesh verification step (S64)

[0282] According to one embodiment of the present invention, in the mesh verification step (S64), the 3D mesh generation module (260) can verify whether the generated mesh is structurally free of defects and evaluate whether the quality of the mesh is suitable for building a high-resolution 3D model. Through this, the 3D mesh generation module (260) can detect defective meshes and identify areas requiring modification. Specifically, the 3D mesh generation module (260) can utilize an Autoencoder model (S64-1) and a Pretrained Classifier model (S64-2) for mesh verification.

[0283] According to one embodiment of the present invention, the Autoencoder model (S64-1) is a deep learning-based unsupervised learning model that can learn the main features of input data through the process of compressing (encoding) and restoring (decoding) input data. Specifically, the 3D mesh generation module (260) can detect structural defects in the mesh to be verified by utilizing the Autoencoder model (S64-1) to learn the pattern of a normal mesh structure. Specifically, the 3D mesh generation module (260) calculates a reconstruction error during the process of restoring the input mesh using the Autoencoder model (S64-1), and can consider areas with large errors as defective meshes.

[0284] According to one embodiment of the present invention, the Pretrained Classifier model (S64-2) is a pre-trained classification model that can perform the role of distinguishing between normal meshes and defective meshes. Specifically, the 3D mesh generation module (260) can utilize the Pretrained Classifier model (S64-2) to learn the characteristics of normal / abnormal meshes using a large-scale mesh dataset, and then evaluate the input mesh in the verification step to determine whether it is defective. In addition, the 3D mesh generation module (260) can utilize the Pretrained Classifier model (S64-2) to evaluate the quality of the mesh based on the size, angle, ratio of each triangle, and connectivity between triangles. Through this, the 3D mesh generation module (260) can quickly and accurately detect defective meshes. Furthermore, the 3D mesh generation module (260) can utilize the Pretrained Classifier model (S64-2) to classify defect types such as holes, distortions, and duplicate triangles, and provide specific information for correction work.

[0285] Detail addition step (S65)

[0286] According to one embodiment of the present invention, in the detail addition step (S65), the 3D mesh generation module (260) can perform the task of building a realistic and sophisticated high-resolution 3D model by adding more details to the generated 3D mesh. Specifically, the 3D mesh generation module (260) can enhance the visual quality by enhancing the texture, pattern, and fine structure of the mesh surface.

[0287] According to one embodiment of the present invention, a 3D mesh generation module (260) may utilize a Neural Style Transfer model (S65-1) used to learn a style, such as a pattern or texture, from one image to add detail and apply it to another image or surface. Specifically, the 3D mesh generation module (260) may utilize the Neural Style Transfer model (S65-1) to apply style information learned from reference data, such as satellite images or surface textures, to the 3D mesh surface. Additionally, the 3D mesh generation module (260) may utilize the Neural Style Transfer model (S65-1) to enhance the patterns and textures of the mesh surface, thereby adding detailed visual characteristics such as mountainous terrain, building exteriors, and road surfaces. That is, the 3D mesh generation module (260) can represent the mesh surface more realistically and precisely.

[0288] According to one embodiment of the present invention, a 3D mesh generation module (260) can generate and add missing details based on existing mesh surface data by utilizing a Generative Adversarial Networks (GAN) model (S65-2). For example, the 3D mesh generation module (260) can reinforce the surface by utilizing a GAN model (S65-2) to generate detailed structures such as fine textures, leaf patterns, road cracks, and brick patterns on the mesh surface. Additionally, the 3D mesh generation module (260) can improve visual quality by utilizing a GAN model (S65-2) to add natural textures and patterns learned from actual satellite image data to the mesh surface.

[0289] Texture mapping step (S66)

[0290] According to one embodiment of the present invention, in the texture mapping step (S66), the 3D mesh generation module (260) can generate a more realistic and visually complete 3D model by combining textures obtained from optical images (RGB), SAR (radar images), or other data with the generated 3D mesh. Specifically, the cGAN (Conditional Generative Adversarial Networks) model (S66-1) is a deep learning model used for texture generation and mapping, and can generate textures based on specific conditions such as DEM and RGB image data. Specifically, the 3D mesh generation module (260) can utilize the cGAN model (S66-1) to receive DEM data and optical image data as inputs and generate a texture suitable for the surface of the 3D mesh. For example, the 3D mesh generation module (260) can utilize the cGAN model (S66-1) to learn building elevation information from the DEM and the color and texture of the building from the optical image, and generate a texture by combining each data. Accordingly, a DEM-based 3D mesh can be naturally combined with an optical image.

[0291] According to one embodiment of the present invention, the Deep Texture Networks model (S66-2) can perform the role of generating a high-resolution texture and mapping it to the surface of a 3D mesh. Accordingly, the 3D mesh generation module (260) can utilize the Deep Texture Networks model (S66-2) to receive DEM, optical images, and SAR data as input, analyze the key feature points of each data, and generate a detailed texture based thereon. Specifically, the 3D mesh generation module (260) can utilize the Deep Texture Networks model (S66-2) to automatically perform UV mapping between the surface of the 3D mesh and the input texture, and calculate UV coordinates to minimize distortion of the texture and improve mapping accuracy.

[0292] According to one embodiment of the present invention, in the texture mapping step (S66), the 3D mesh generation module (260) can generate a more realistic 3D model by combining optical images (RGB) and SAR (radar images) with a DEM (Digital Elevation Model). At this time, when combining optical images, etc., with a 3D object generated based on the DEM, the 3D mesh generation module (260) can automatically perform accurate UV mapping using a pre-model trained through a CNN (Convolutional Neural Network). Specifically, for alignment between the optical images and the DEM, the CNN-based pre-model can detect corner points or boundary lines of a building. In this process, the 3D mesh generation module (260) can perform alignment by utilizing the SIFT (Scale-Invariant Feature Transform) algorithm to detect points where pixel values ​​change rapidly in the case of optical images, and corner points where elevation changes are distinct in the case of DEM. In addition, the 3D mesh generation module (260) can detect key feature points such as buildings, roads, and terrain boundaries in the case of SAR images, and align them with DEM and optical images through a CNN model to finally perform accurate texture mapping.

[0293] Lighting and material setting step (S67)

[0294] According to one embodiment of the present invention, in the lighting and material setting step (S67), the 3D mesh generation module (260) can generate a realistic and sophisticated 3D model by adding realistic lighting effects and material expressions to the generated 3D mesh. Specifically, the NeRF (Neural Radiance Fields) model (S67-1) is a technology that utilizes deep learning to learn lighting, color, and material information within a 3D space and generates realistic visual expressions based thereon, and the 3D mesh generation module (260) can utilize the NeRF model (S67-1).

[0295] According to one embodiment of the present invention, a 3D mesh generation module (260) can learn lighting information to be applied to a 3D mesh by utilizing a NeRF model (S67-1). At this time, the 3D mesh generation module (260) can estimate the position, direction, and intensity of a light source in space by utilizing a 3D model and multi-angle optical and SAR image data as input data. Accordingly, the learned lighting model can generate realistic lighting effects by calculating how the light source is reflected and absorbed on the mesh surface. Through this, the 3D mesh generation module (260) can express highlight effects or shadow effects where the light is reflected on the mesh surface according to the angle of incidence.

[0296] According to one embodiment of the present invention, the 3D mesh generation module (260) can learn surface materials such as metal, wood, and concrete of the 3D mesh by utilizing the NeRF model (S67-1) and simulate the interaction of light sources by reflecting the characteristics of these materials. That is, the 3D mesh generation module (260) can realistically represent the physical characteristics of the mesh by adjusting the degree of light absorption, scattering, and reflection according to the material. For example, the 3D mesh generation module (260) can simulate light transmission and refraction effects on glass surfaces and strong light reflection on metal surfaces by utilizing the NeRF model (S67-1).

[0297] According to one embodiment of the present invention, a 3D mesh generation module (260) can utilize a NeRF model (S67-1) to convert a 3D mesh into volume data and render it in a manner where lighting and materials interact in a real environment. At this time, the 3D mesh generation module (260) can determine how the mesh appears from various viewing angles by calculating the relationship between the position of each pixel and the light source. Additionally, the 3D mesh generation module (260) can utilize a NeRF model (S67-1) to adjust the visual representation of the mesh not only in a single light source environment but also in a multi-light source environment. Specifically, when there are multiple light sources, the 3D mesh generation module (260) can provide natural results even in complex lighting environments by calculating the light trajectories and reflection effects generated from each light source.

[0298] FIG. 13 is a drawing showing a digital elevation model generated according to one embodiment of the present invention.

[0299] As illustrated, a Digital Surface Model (DSM) can represent 3D data generated based on satellite imagery or aerial data on a 2D plane. Specifically, bright areas may represent relatively high altitudes, while dark areas may represent low altitudes. Additionally, the illustrated Y-shaped structure may represent major buildings and may be generated based on the building's outline and height in the DSM data, while dark areas may represent flat land or lowlands and may contrast with areas representing high altitudes.

[0300] FIG. 14 is a drawing showing a converted point cloud model according to one embodiment of the present invention.

[0301] As described above, the point cloud model is a three-dimensional data structure generated based on a digital elevation model (DSM), and each point can represent 3D coordinates calculated based on satellite imagery or aerial data. That is, each point represents a specific location on the actual terrain or the surface of an object such as a building, road, or tree, and it can be seen that the point cloud model visually represents three-dimensional data to indicate the height and depth of the building and terrain. As described above, a point cloud model such as that shown in FIG. 14 can be generated by a point cloud generation module (230). Specifically, the point cloud generation module (230) can generate a point cloud model by calculating the location of each point based on X, Y, and Z coordinates extracted from the digital elevation model, performing stereo matching using two or more satellite images or aerial photographs, and calculating the altitude (Z value) of each point.

[0302] FIG. 15 is a drawing showing the final generated result according to one embodiment of the present invention.

[0303] As illustrated, FIG. 15 is a finally generated 3D mesh model, representing a high-resolution 3D structure generated based on a digital elevation model (DSM) and point cloud data.

[0304] FIG. 16 is a diagram showing the steps for performing an algal bloom detection system according to one embodiment of the present invention.

[0305] In this specification, the term "module" does not refer to a specific physical component but rather to a software tool implemented in a memory unit within one or more devices. The processor of each device performs a designated function by reading and executing the corresponding tool implemented in the memory unit, and these functional units of tools are referred to as modules for convenience. Accordingly, the algal bloom detection system of the present invention may include not only a satellite image input device but also one or more other devices, and the entire configuration is referred to as the algal bloom detection system.

[0306] RGB channel optical satellite image input step (S1000)

[0307] According to one embodiment of the present invention, the satellite image input device (1100) of the algal bloom detection system (1000) can perform the role of receiving satellite image data provided from an external source. Specifically, the satellite image input device (1100) can perform the role of receiving raw satellite image data (3a-1, raw satellite imagery) captured from a satellite. Accordingly, the memory unit (1700) of the algal bloom detection system (1000) can store the raw satellite image data (3a-1) received by the satellite image input device (1100). At this time, the display unit (1800) of the algal bloom detection system (1000) can perform the role of providing the input raw satellite image data (3a-1) to the user so that the user can visually check it. Accordingly, the user can verify the raw satellite image data (3a-1), which is the input data of the algal bloom detection system (1000), or check whether there are any abnormalities in the initial data.

[0308] Each component of the algae detection system (1000) described in this specification, such as a display unit (1800), does not necessarily mean that it is independently provided throughout the entire system. That is, the display unit (1800) may be a display unit included in any one of the one or more devices constituting the algae detection system (1000), and may allow a user to check the algae detection results through the screen of the device. As such, each module or component described in this specification may be an element mounted on some of the multiple devices constituting the system, and is described as a component of the entire system for convenience.

[0309] Image preprocessing step (S2000)

[0310] RGB band integration step (S2100)

[0311] According to one embodiment of the present invention, the image processing module (1200) of the algal bloom detection system (1000) can separate the R (red) channel, G (green) channel, and B (blue) channel from the raw satellite image data (3a-1) received by the satellite image input device (1100) and stored in the memory unit (1700). Specifically, the raw satellite image data (3a-1) may not be a simple RGB image, but may include multispectral data or be provided in a form where color and brightness are distorted. In addition, the raw satellite image data (3a-1) may have unbalanced color information depending on environmental conditions such as lighting and atmospheric conditions in each region or the characteristics of the satellite sensor. Therefore, the image processing module (1200) can separate the raw satellite image data (3a-1) into the R (red) channel, G (green) channel, and B (blue) channel, independently analyze and adjust the characteristics of each channel, and then perform integration.

[0312] Here, the image processing module (1200) does not refer to a physical device independently provided throughout the entire algae detection system (1000), but rather to a software tool installed in the memory section of one or more devices constituting the algae detection system (1000). The processor of the device executes this tool stored in the memory section to perform image processing functions, and for convenience, it is referred to as the image processing module (1200). Accordingly, the image processing module (1200) described in this specification may also be a functional element included within a specific device and is merely described as a component of the entire system.

[0313] According to one embodiment of the present invention, the image processing module (1200) of the green algae detection system (1000) can separate the R (red) channel, G (green) channel, and B (blue) channel from raw satellite image data (3a-1), adjust and correct the brightness and color balance for each channel, and then perform integration. Specifically, since the original color information of the satellite image may be distorted due to the influence of atmospheric conditions, light reflection, sensor noise, etc., the image processing module (1200) can generate RGB image data optimized for detection in which the color information is not distorted.

[0314] According to one embodiment of the present invention, the most important information for detecting algal blooms is mainly contained in the G (green) channel; however, if only green information is analyzed, it may be difficult to distinguish it from other elements such as water reflections and shadows. Accordingly, the image processing module (1200) can separate the R (red) channel, G (green) channel, and B (blue) channel to emphasize the data in the G (green) channel, or to further analyze specific bands and then integrate them to emphasize the green information while preserving the overall image information. In addition, since raw satellite image data (3a-1) may contain noise such as noise, the image processing module (1200) can separate the R (red) channel, G (green) channel, and B (blue) channel to perform noise removal and filtering operations on each channel, and then integrate each channel to improve the quality of the image. That is, the image processing module (1200) can provide the effect of increasing the accuracy of segmentation or clustering operations in subsequent steps.

[0315] Required area division step (S2200)

[0316] According to one embodiment of the present invention, the first water area data may refer to data regarding a water area generated by an image processing module (1200) distinguishing between a water area and a non-water area through a segmentation technique. Additionally, the first water area data may refer to sea area data generated by a segmentation module (1300) initially distinguishing between land and sea based on image data separated into R, G, and B channels. On the other hand, the second water area data may refer to data in which the segmentation module (1300) performs additional segmentation operations based on the first water area data to further subdivide the interior of the water area. That is, the second water area data may refer to data resulting from distinguishing between areas with algal blooms and areas without algal blooms within the water area, or dividing the boundaries within the water area more precisely.

[0317] According to one embodiment of the present invention, an image processing module (1200) of an algae detection system (1000) can distinguish between a water body containing first water body data and a non-water body through a segmentation technique such as filtering a specific color or brightness range. For example, the image processing module (1200) can identify a water body, which is a required area, through color, brightness, and texture-based filtering based on data after noise removal and filtering operations have been performed in each channel. Accordingly, the image processing module (1200) can extract first water body data, and the image processing module (1200) can use the image data containing first water body data as input data for a subsequent step, a deep learning-based segmentation execution step (S3100).

[0318] Machine learning stage (S3000)

[0319] Deep learning-based segmentation execution step (S3100)

[0320] According to the present invention, segmentation refers to the process of analyzing an image at the pixel level to separate specific regions or features. That is, it may refer to the process of predicting which class or group each pixel of an image belongs to, and thereby dividing the image into detailed parts. In the algal bloom detection system (1000) of the present invention, the segmentation module (1300) can perform the role of distinguishing between water bodies such as seas, rivers, and lakes and land in satellite images. Additionally, the segmentation module (1300) can perform the role of distinguishing between areas with algal blooms and areas without algal blooms within a water body at the pixel level. Furthermore, since the deep learning-based segmentation utilized by the segmentation module (1300) has high precision, the present invention can provide the effect of performing precise green area detection.

[0321] Here, the segmentation module (1300) does not refer to a physical device independently provided for the entire algae detection system (1000), but rather to a software tool installed in a memory unit within one or more devices constituting the system. The processor of the device executes this tool stored in the memory unit to perform a segmentation function, and for convenience, this functional unit tool is referred to as the segmentation module (1300). Accordingly, the segmentation module (1300) described in this specification may be a functional element included within a specific device, and is expressed as a component of the entire system for convenience of explanation.

[0322] According to one embodiment of the present invention, the first water area data may refer to data regarding a water area generated by an image processing module (1200) distinguishing between a water area and a non-water area through a segmentation technique. Additionally, the first water area data may refer to sea area data generated by a segmentation module (1300) initially distinguishing between land and sea based on image data separated into R, G, and B channels. On the other hand, the second water area data may refer to data in which the segmentation module (1300) performs additional segmentation operations based on the first water area data to further subdivide the interior of the water area. That is, the second water area data may refer to data resulting from distinguishing between areas with algal blooms and areas without algal blooms within the water area, or dividing the boundaries within the water area more precisely.

[0323] According to one embodiment of the present invention, the segmentation module (1300) can distinguish detailed areas within a water body on a pixel-by-pixel basis in image data containing first water body data by applying deep learning-based algorithms such as U-Net, Fully Convolutional Network, and DeepLab. Specifically, the segmentation module (1300) can predict the class of each pixel in the image data containing first water body data by utilizing a pre-trained deep learning model. For example, the segmentation module (1300) can perform a prediction on whether each pixel in the image data containing first water body data corresponds to an algae bloom area, a water area, or other areas by utilizing a pre-trained deep learning model.

[0324] Additionally, the segmentation module (1300) can distinguish between areas suspected of having algal blooms and areas without algal blooms within the water body, and can also distinguish the boundaries within the water body. Consequently, the segmentation module (1300) can generate second water body data after performing segmentation using image data containing first water body data as input. That is, the segmentation module (1300) performs a prediction on whether each pixel corresponds to an algal bloom area, a water area, or other areas, and can classify pixels predicted to correspond to an algal bloom area as second water body data.

[0325] According to one embodiment of the present invention, the memory unit (1700) can store intermediate data and final segmentation results generated during a deep learning segmentation operation by the segmentation module (1300). Additionally, the memory unit (1700) may perform the role of storing the segmentation result, which is a separated region mask, and transmitting it to the clustering module (1400). That is, the final segmentation result stored in the memory unit (1700) may represent the second water body data.

[0326] K means clustering algorithm application step (S3200)

[0327] According to the present invention, a cluster may refer to a set of data points having similar characteristics in data analysis. Here, clustering may refer to the process of dividing data into several groups of clusters. That is, each cluster is distinct from other clusters, but the data within a cluster may share similar characteristics. Accordingly, the algal bloom detection system (1000) of the present invention can group data to make it easier to understand, find patterns, and utilize them for analysis.

[0328] Specifically, the clustering module (1400) can set the number of clusters (k) small if the color distribution within the water body is relatively simple, and set the number of clusters (k) large if the water body is wide and the color diversity is large, resulting in a relatively complex color distribution within the water body. That is, the clustering module (1400) can experimentally determine the optimal k value by analyzing the data. Additionally, K-means Clustering is an unsupervised learning algorithm that divides data into k clusters. The clustering module (1400) can receive data to be analyzed, set k initial centroids, assign each pixel data to the cluster closest to the centroid based on the distance from the centroid, calculate the average value of each cluster, and perform a process of updating the centroids. Accordingly, the clustering module (1400) can perform grouping of pixels with similar characteristics by repeating the above process until the centroids no longer change.

[0329] According to one embodiment of the present invention, the clustering module (1400) can execute a K-Means Clustering algorithm to group the second body data generated by the segmentation module (1300) into major color clusters. Here, each cluster may refer to a set of pixels having similar colors. Specifically, the clustering module (1400) may receive color data based on the second body data generated by the segmentation module (1300). At this time, the clustering module (1400) may classify the pattern of the data by setting an appropriate number of clusters (k) as a parameter of the algorithm. For example, a bright blue cluster may be classified as shallow water, a dark blue cluster as deep water, a dark green cluster as algae bloom, and a turbid gray cluster as floating matter.

[0330] Algal bloom area detection mask generation step (S4000)

[0331] According to one embodiment of the present invention, the clustering module (1400) can select a cluster related to algal bloom. Accordingly, the algal bloom detection mask generation module (1500) can perform binarization on the cluster selected by the clustering module (1400). Specifically, the algal bloom detection mask generation module (1500) can set the pixel values ​​belonging to the selected cluster corresponding to the area where the algal bloom is located to white, and the remaining pixel values ​​corresponding to the area where the algal bloom is not located to black. That is, the algal bloom detection mask generation module (1500) can generate a detection mask composed of 0s and 1s through binarization processing. Here, the detection mask is data that highlights only the area where algal bloom is detected, and may refer to binarized data separated into a white algal bloom area and a black non-algal bloom area. That is, the detection mask can be represented as the algal bloom detection mask generation image (3b-3) shown in FIG. 18b.

[0332] Here, the clustering module (1400) and the algae detection mask generation module (1500) do not refer to independent hardware devices, but rather to software tools installed in the memory section of one or more devices constituting the algae detection system (1000). The processor of the device executes these tools stored in the memory section to perform clustering and mask generation functions, and such functional units of tools are referred to as modules for convenience. Accordingly, the clustering module (1400) and the algae detection mask generation module (1500) described in this specification may be functional elements included within a specific device, and are merely expressed as components of the overall system for the convenience of explanation.

[0333] According to one embodiment of the present invention, a clustering module (1400) is configured to select a cluster related to algal bloom among the generated clusters, and an algal bloom detection mask generation module (1500) is configured to perform binarization on the selected cluster. Additionally, the algal bloom detection mask generation module (1500) is configured to set pixel values ​​belonging to the cluster selected by the clustering module (1400) to white, and the remaining pixel values ​​not belonging to the selected cluster to black.

[0334] According to one embodiment of the present invention, the algae detection mask generation module (1500) may remove noise generated in the K means clustering algorithm application step (S3200) and smooth the boundary lines. Specifically, the algae detection mask generation module (1500) may perform morphological operations to manipulate the shape of an object by changing, expanding, or shrinking the pixel values ​​in a binarized image. Additionally, the algae detection mask generation module (1500) may refine the mask using a filtering technique and exclude meaningless data by removing clusters or pixel groups that are too small.

[0335] Strength calculation step (S5000)

[0336] According to one embodiment of the present invention, the intensity calculation and visualization module (1600) can quantitatively calculate the intensity of an algal bloom area based on a detection mask generated by the algal bloom detection mask generation module (1500). Specifically, the intensity calculation and visualization module (1600) can determine the distribution and size of the algal bloom by calculating the ratio of white pixels representing an algal bloom area and black pixels representing a non-algal bloom area in the detection mask. At this time, the intensity of the algal bloom area is calculated by comparing the number of white pixels with the total number of pixels. For example, if white pixels account for 30% of the total pixels in the detection mask, this means that the algal bloom occupies 30% of the area.

[0337] According to one embodiment of the present invention, the intensity calculation and visualization module (1600) calculates a more precise intensity value by utilizing color and brightness information in addition to such pixel-based calculations. In particular, since the value of the G (green) channel in RGB data provides important information for detecting algal blooms, the concentration of algal blooms can be calculated based on at least one of the average and sum of the G channel values. For example, the intensity calculation and visualization module (1600) can determine that areas with high G channel values ​​are areas with dense algal blooms, and areas with low G channel values ​​are areas with relatively little or no algal blooms.

[0338] According to one embodiment of the present invention, the intensity calculation and visualization module (1600) can more accurately estimate the concentration of algal bloom by correcting environmental factors such as color distortion and brightness change through an intensity calculation algorithm. At this time, the intensity calculation and visualization module (1600) can utilize not only the G channel value but also the R (red) and B (blue) channel values ​​as auxiliary tools to minimize the influence of other elements within the water body, such as shadows and water reflections, on the calculation of algal bloom intensity.

[0339] According to one embodiment of the present invention, the intensity calculation and visualization module (1600) can store the calculated intensity value in the memory unit (1700). The intensity value stored in the memory unit (1700) can be converted into a heatmap, graph, color map, etc., during the subsequent visualization process. In addition, the intensity calculation and visualization module (1600) can provide data to the user that allows them to intuitively check the concentration, distribution, and intensity changes of the algal bloom based on this. For example, when the calculated intensity value is expressed as a heatmap, areas with dense algal bloom are displayed in red, and areas with low concentration are displayed in green, which provides the effect of enabling the user to quickly grasp the severity of the algal bloom.

[0340] Mask Integration and Visualization Step (S6000)

[0341] According to one embodiment of the present invention, the intensity calculation and visualization module (1600) may perform quantification of the severity of the algal bloom based on criteria such as pixel density or color intensity. For example, the intensity calculation and visualization module (1600) may perform quantification by calculating a pixel density of 0% or more and 100% or less, or by calculating a result in ppm (parts per million) units. At this time, the display unit (1800) may perform the role of visually displaying the calculated algal bloom intensity so that the user can easily check it, and may output the intensity calculation result using a visualized heatmap, graph, or numerical value. For example, the display unit (1800) may provide visual materials so that the user intuitively understands the severity of the algal bloom by displaying high-intensity areas in red, low-intensity areas in green, etc.

[0342] Result output step (S7000)

[0343] According to one embodiment of the present invention, the display unit (1800) can output the detected distribution and intensity of the algal bloom in a visual form. Specifically, the display unit (1800) can output visual data such as a heat map, a graph, or a color map, and may also provide a user interface that allows zooming in / out of results or clicking to view detailed information.

[0344] FIG. 17 is a diagram illustrating the configuration of a green algae detection system according to one embodiment of the present invention.

[0345] Satellite image input device (1100)

[0346] As described above, the satellite image input device (1100) of the algae detection system (1000) can perform the role of receiving raw satellite image data captured from an external satellite. Specifically, the received raw data may include multispectral data rather than simple RGB images, or may be provided in a form with distorted color and brightness. That is, the satellite image input device (1100) can store the received raw data in the memory unit (1700) of the algae detection system (1000) and can visually provide the input raw data to the user through the display unit (1800).

[0347] Image processing module (1200)

[0348] As described, the image processing module (1200) of the algal bloom detection system (1000) can perform the role of processing raw satellite image data into a form suitable for detection. Specifically, the image processing module (1200) can separate the R (red), G (green), and B (blue) channels from the raw satellite image data, independently analyze and adjust the characteristics of each channel, and then integrate them to generate an RGB image optimized for detection. That is, the image processing module (1200) can correct color information distorted by atmospheric conditions, light reflection, sensor noise, etc., and improve the quality of the image.

[0349] According to one embodiment of the present invention, the image processing module (1200) can improve data quality by performing noise removal and filtering operations in each channel, and can perform the role of increasing the accuracy of subsequent segmentation and clustering operations. That is, the image processing module (1200) can emphasize green information while preserving the entire image information by emphasizing data of the G (green) channel, which is important in algal bloom detection according to the present invention, or by additionally analyzing a specific band. In addition, the image processing module (1200) can distinguish between water bodies and non-water bodies by applying color, brightness, and texture-based filtering techniques, identify necessary areas, and provide them as input data for a subsequent deep learning-based segmentation operation.

[0350] Segmentation module (1300)

[0351] As described, the segmentation module (1300) of the algae detection system (1000) can perform the role of separating and classifying areas necessary for algae detection by analyzing satellite image data in pixel units. Specifically, the segmentation module (1300) can distinguish between water bodies such as seas, rivers, and lakes and land by applying deep learning-based algorithms such as U-Net, Fully Convolutional Network, and DeepLab, and can precisely distinguish between areas with algae and areas without algae within the water body.

[0352] According to one embodiment of the present invention, the segmentation module (1300) utilizes a pre-trained deep learning model to predict each class, such as algal bloom, normal water, and other areas, on a pixel-by-pixel basis, and enables clear distinction of boundaries within the water body and detection of areas suspected of having algal bloom. Additionally, the segmentation module (1300) can store intermediate data generated during the segmentation process and the final results in a memory unit (1700) and subsequently transmit them to the clustering module (1400).

[0353] Clustering module (1400)

[0354] According to one embodiment of the present invention, the clustering module (1400) of the algal bloom detection system (1000) can group data having similar characteristics based on data generated by the segmentation module (1300). Specifically, the clustering module (1400) can divide the data into multiple clusters by applying a K-means clustering algorithm, set an initial centroid, assign each data point to the nearest cluster, and then repeatedly update the centroid to form an optimal cluster.

[0355] According to one embodiment of the present invention, the clustering module (1400) can distinguish various areas within a body of water based on data such as color, brightness, and texture, such as classifying bright blue as shallow water, dark blue as deep water, dark green as algae, and turbid gray as floating matter. Additionally, the clustering module (1400) can identify clusters related to algae and transmit them to the algae detection mask generation module (1500).

[0356] Algae detection mask generation module (1500)

[0357] As described above, the algae detection mask generation module (1500) of the algae detection system (1000) can perform the role of generating and refining a binarized mask for algae detection based on data transmitted from the clustering module (1400). Specifically, the algae detection mask generation module (1500) can generate a detection mask that distinguishes between algae areas and non-algae areas by setting pixels belonging to clusters related to algae to white and the remaining pixels to black. That is, the algae detection mask generation module (1500) can perform morphological operations on the generated mask to smooth the boundaries of objects and increase the accuracy of the detection results by changing, expanding, or shrinking pixel values.

[0358] According to one embodiment of the present invention, the algae detection mask generation module (1500) can reduce noise by removing small clusters or meaningless pixel groups and refine the detection results by applying additional filtering techniques. Accordingly, the finally generated detection mask is transmitted to the intensity calculation and visualization module (1600) and can be used as basic data for calculating and visualizing algae intensity.

[0359] Strength calculation and visualization module (1600)

[0360] As described above, the intensity calculation and visualization module (1600) of the algae detection system (1000) can perform the role of calculating the intensity of the algae area based on data provided by the algae detection mask generation module (1500), quantifying it, and visually representing it. Specifically, the intensity calculation and visualization module (1600) can determine the size and distribution of the algae area by calculating the number of white pixels in the detection mask, and can accurately estimate the concentration of the algae using the average or sum of the G (green) channel values. At this time, the calculated algae intensity is quantified in units such as pixel density, ratio (0% or more and 100% or less), or ppm, and the intensity calculation and visualization module (1600) can store the quantified data in the memory unit (1700).

[0361] According to one embodiment of the present invention, the intensity calculation and visualization module (1600) can express the intensity calculation results in various visual materials such as heatmaps, graphs, and color maps. Specifically, the intensity calculation and visualization module (1600) displays high-intensity areas in red and low-intensity areas in green so that the user can intuitively understand the severity of the algal bloom. In addition, the intensity calculation and visualization module (1600) may provide a user interface that allows the user to zoom in / out on the results or explore detailed information.

[0362] Memory unit (1700)

[0363] As described above, the memory unit (1700) of the algae detection system (1000) can perform data storage within the system. Specifically, the memory unit (1700) stores raw satellite image data received by the satellite image input device (1100) to prepare it for use in subsequent processing steps, and stores intermediate data generated in each processing step, such as the image processing module (1200), segmentation module (1300), and clustering module (1400), to enable subsequent operations. Additionally, the memory unit (1700) stores the final detection mask generated by the algae detection mask generation module (1500) and the algae intensity calculation results generated by the intensity calculation and visualization module (1600), so that they can be used for visualization and analysis.

[0364] Display unit (1800)

[0365] As described above, the display unit (1800) of the algae detection system (1000) can perform the role of outputting information and providing an intuitive interface so that the user can visually check the input data and processing results of the system. Specifically, the display unit (1800) can visually provide raw satellite image data received from the satellite image input device (1100). In addition, the display unit (1800) visually displays data processed in each module, such as segmentation results, clustering results, algae detection masks, intensity calculation results, etc., so that the user can intuitively understand the data.

[0366] According to one embodiment of the present invention, the display unit (1800) visualizes the intensity calculation results in various forms such as a heatmap, graph, and color map, and high-intensity areas are displayed in red and low-intensity areas in green so that the severity of the algal bloom can be easily identified. In addition, the display unit (1800) provides a user interface that allows zooming in / out and checking detailed information, enabling efficient data exploration and analysis, and can also perform the function of notifying the user through a visual warning when a severe algal bloom exceeding a threshold is detected.

[0367] FIG. 18a is a diagram illustrating an embodiment for separating a detection target area using raw satellite image data according to the present invention.

[0368] As described, raw satellite image data (3a-1) refers to unprocessed raw image data and may be image data in a state where data captured by a satellite camera or sensor is directly collected. Specifically, raw satellite image data (3a-1) may include spectral data of various bands, such as infrared (NIR) and near-infrared, as well as RGB (red, green, blue) channels, and may have high resolution according to satellite imaging technology. That is, raw satellite image data (3a-1) may have a precision of 10 m / pixel or 1 m / pixel, thereby allowing for the identification of minute changes in water or land. However, raw satellite image data (3a-1) may contain distortion or noise due to the atmosphere, clouds, solar reflection, sensor errors, etc.

[0369] As described above, RGB channel data separation (3a-2) may refer to the process of individually extracting each spectral band (channel). Specifically, RGB channel data separation (3a-2) may be performed by the image processing module (1200) of the green algae detection system (1000). As described above, the image processing module (1200) may separate the R (red) channel, G (green) channel, and B (blue) channel from the raw satellite image data (3a-1) received by the satellite image input device (1100) and stored in the memory unit (1700). Specifically, the raw satellite image data (3a-1) may not be a simple RGB image, but may include multispectral data or be provided in a form where color and brightness are distorted. Additionally, the raw satellite image data (3a-1) may have unbalanced color information depending on environmental conditions such as lighting and atmospheric conditions in each region or the characteristics of the satellite sensor. Accordingly, the image processing module (1200) can separate the raw satellite image data (3a-1) into R (red) channel, G (green) channel, and B (blue) channel, independently analyze and adjust the characteristics of each channel, and then perform integration.

[0370] According to one embodiment of the present invention, in the RGB channel data separation (3a-2), the R (red) channel may be a channel that emphasizes the reflection characteristics and contrast of the water body, and the G (green) channel may be a channel containing information that is significant for detecting algal blooms, as algal blooms mainly have high values ​​in the green band. Additionally, since the B (blue) channel may be a channel used to analyze the depth and reflectance of the water, the image processing module (1200) can separate each channel and analyze it independently, and then integrate them if necessary for use in detecting algal blooms.

[0371] As described above, RGB channel data integration (3a-3) may be a process of generating multidimensional image data by reintegrating each spectral channel separated through RGB channel data separation (3a-2). As previously mentioned, the image processing module (1200) of the green algae detection system (1000) may separate the R (red) channel, G (green) channel, and B (blue) channel from the raw satellite image data (3a-1), adjust and correct the brightness and color balance for each channel, and then perform integration. Specifically, since the original color information of the satellite image may be distorted due to the influence of atmospheric conditions, light reflection, sensor noise, etc., the image processing module (1200) may generate RGB image data optimized for detection in which the color information is not distorted.

[0372] According to one embodiment of the present invention, the most important information for detecting algal blooms is mainly contained in the G (green) channel; however, if only green information is analyzed, it may be difficult to distinguish it from other elements such as water reflections and shadows. Accordingly, the image processing module (1200) can separate the R (red) channel, G (green) channel, and B (blue) channel to emphasize the data in the G (green) channel, or to further analyze specific bands and then integrate them to emphasize the green information while preserving the overall image information. In addition, since raw satellite image data (3a-1) may contain noise such as noise, the image processing module (1200) can separate the R (red) channel, G (green) channel, and B (blue) channel to perform noise removal and filtering operations on each channel, and then integrate each channel to improve the quality of the image. That is, the image processing module (1200) can provide the effect of increasing the accuracy of segmentation or clustering operations in subsequent steps.

[0373] As described, the separation of the detection target area (3a-4) may be a task of separating necessary areas, such as water bodies for algal bloom detection, from unnecessary areas, such as land and cities, based on integrated satellite image data. That is, the image processing module (1200) of the algal bloom detection system (1000) can distinguish between water bodies and non-water bodies through segmentation techniques, such as filtering specific color or brightness ranges. For example, the image processing module (1200) can identify water bodies, which are necessary areas, through color, brightness, and texture-based filtering based on data after noise removal and filtering operations have been performed in each channel. Accordingly, the image processing module (1200) can extract first water body data, and the image processing module (1200) can use the image data containing the first water body data as input data for the subsequent step of deep learning-based segmentation execution (S3100). Accordingly, the present invention can provide the effect of reducing errors in the algal bloom detection process and increasing the reliability of results by distinguishing between the detection target area and the non-target area.

[0374] According to the present invention, segmentation refers to the process of analyzing an image at the pixel level to separate specific regions or features. That is, the separation of the detection target area (3a-4) may refer to the process of predicting which class or group each pixel of an image belongs to, and thereby dividing the image into detailed parts. In the algal bloom detection system (1000) of the present invention, the segmentation module (1300) can perform the role of separating the detection target area (3a-4) by distinguishing between water bodies such as seas, rivers, and lakes and land in satellite images. Additionally, the segmentation module (1300) can perform the role of separating the detection target area (3a-4) by distinguishing between areas with algal blooms and areas without algal blooms at the pixel level within the water body. Furthermore, since the deep learning-based segmentation utilized by the segmentation module (1300) has high precision, the present invention can provide the effect of performing precise green area detection.

[0375] According to one embodiment of the present invention, the segmentation module (1300) can distinguish detailed areas within a water body on a pixel-by-pixel basis in image data containing the first water body data by applying deep learning-based algorithms such as U-Net, Fully Convolutional Network, and DeepLab. Additionally, the segmentation module (1300) can separate areas of the image data containing the first water body data by using Thresholding, Edge Detection, or Clustering algorithms. Specifically, the segmentation module (1300) can predict the class of each pixel of the image data containing the first water body data by utilizing and applying a pre-trained deep learning model. For example, the segmentation module (1300) can perform a prediction on whether each pixel of the image data containing the first water body data corresponds to green algae, normal water, or other areas by utilizing a pre-trained deep learning model. Additionally, the segmentation module (1300) can distinguish between areas suspected of having green algae and areas without green algae within the water body, and can also distinguish the boundaries within the water body. Consequently, the segmentation module (1300) can generate second water body data after performing segmentation using image data containing first water body data as input.

[0376] According to one embodiment of the present invention, the memory unit (1700) can store intermediate data and final segmentation results generated during a deep learning segmentation operation by the segmentation module (1300). Additionally, the memory unit (1700) may perform the role of storing the segmentation result, which is a separated region mask, and transmitting it to the clustering module (1400). That is, the final segmentation result stored in the memory unit (1700) may represent the second water body data.

[0377] FIG. 18b is a drawing for explaining an embodiment of calculating and visualizing algal bloom intensity by generating an algal bloom detection mask using actual images according to the present invention.

[0378] As described above, the reference image (3b-1) is the result of performing RGB channel data separation (3a-2) on raw satellite image data (3a-1) and then performing RGB channel data integration (3a-3). It may be an image with improved quality after performing noise removal and filtering operations on each channel in the RGB band integration step (S2100) and then integrating each channel. That is, the reference image (3b-1) may be data that serves as a standard for analysis in subsequent detection operations such as K-means clustering and segmentation performed in the deep learning-based segmentation execution step (S3100) and the K-means clustering algorithm application step (S3200) of the machine learning step (S3000). Additionally, the reference image (3b-1) in which the separated RGB channel data is integrated is in a form that is understandable to humans, allowing the user to visually confirm the location and characteristics of the detection target, so the user can visually distinguish bodies of water such as seas and rivers through the reference image (3b-1). In addition, the reference image (3b-1) is in the form of a color image composed mainly of RGB channels, and each channel can provide independent information.

[0379] As described, the color cluster generation image (3b-2) may refer to an image generated by grouping pixels with similar colors or characteristics based on the reference image (3b-1) as a result of applying the K-means Clustering algorithm. That is, the color cluster generation image (3b-2) may be an image in which each cluster is represented by a specific color so that meaningful patterns, such as green algae areas and normal water bodies, can be visually distinguished in the reference image (3b-1). Here, the clustering module (1400) can perform the K-means Clustering algorithm to group the data of the reference image (3b-1) into clusters, and after analyzing the RGB values ​​of each pixel, can generate k clusters by grouping similar values. That is, as illustrated in FIG. 18b, the reference image (3b-1) is separated into k groups, and in the color cluster generation image (3b-2), bright colors represent normal water bodies or non-green areas, dark green colors represent areas with a high probability of green, and other colors represent areas that are not subject to detection, such as land or cities.

[0380] FIG. 19 is a drawing for explaining an embodiment of combining algal bloom intensity information with a real image using a real image according to the present invention.

[0381] As described, the reference image (4-1) is an image showing raw satellite image data received from the satellite image input device (1100), which is initial data for detecting algal blooms. This image shows the appearance of the actual environment, including water and non-water areas.

[0382] As described, the segmentation technique applied image (4-2) represents the result generated through the segmentation module (1300), distinguishing between water bodies (white) and non-water bodies (black) based on data analyzed at the pixel level. Specifically, the segmentation technique applied image (4-2) can be used as basic data to identify detailed areas within the water bodies.

[0383] As described, the algal bloom intensity calculation and visualization image (4-3) is a result generated by the algal bloom detection mask generation module (1500) and the intensity calculation and visualization module (1600), and is an image that visually represents the quantified algal bloom intensity. Specifically, in the algal bloom intensity calculation and visualization image (4-3), areas with high algal bloom intensity are displayed in yellow or bright colors, and areas with low intensity are displayed in dark colors such as purple, allowing for an intuitive understanding of the concentration distribution of the algal bloom.

[0384] As described above, the combined image of algal bloom intensity information and reference (4-4) is a result generated by combining the algal bloom intensity information with the reference image (4-1), and by overlaying data in which the algal bloom intensity is visualized based on the appearance of the actual environment, the distribution and severity of the algal bloom can be intuitively conveyed to the user. In other words, the present invention provides the effect of being able to analyze algal bloom detection results by connecting them to the actual environment.

[0385] The scope of the present invention is not limited to the embodiments described above but may be implemented in various forms of embodiments within the scope of the appended claims. It is deemed that the scope of the claims of the present invention includes various modifications that are possible by anyone with ordinary knowledge in the technical field to which the invention pertains, without departing from the essence of the invention claimed in the claims.

Claims

1. In a signal processing device for restoring onboard SAR images, RF transceiver for transmitting and receiving analog signals to and from a target; A navigation sensor unit that senses navigation data of the above signal processing device; A signal processing unit that converts an analog signal received by the above RF transceiver into a digital signal; Memory section; and Includes a processor unit; and The above memory unit It is configured to store the digital signal converted by the signal processing unit and the sensed navigation data, and The above processor unit is configured to generate SAR images based on the stored digital signals and navigation data, Signal processing device for restoring onboard SAR images.

2. In Paragraph 1, The above processor unit is, A digital signal stored in the above memory unit is configured to perform signal processing operations of distance compression or directional compression, Signal processing device for restoring onboard SAR images.

3. In Paragraph 1, It further includes a data storage unit; and The above data storage unit Configured to permanently store digital signals and navigation data stored in the above memory unit, Signal processing device for restoring onboard SAR images.

4. In Paragraph 1, The above signal processing unit It further includes an ADC (Analog-to-Digital Converter) module, and The above ADC module A device configured to convert an analog signal received by the above RF transceiver into a digital signal. Signal processing device for restoring onboard SAR images.

5. In Paragraph 1 The above memory unit is, Configured to store the converted digital signal and the sensed navigation data based on the same time, Signal processing device for restoring onboard SAR images.

6. In Paragraph 3, The above memory unit includes DDR memory, and The above data storage unit is composed of an SSD Configured to temporarily and permanently store the above digital signal and the above navigation data, Signal processing device for restoring onboard SAR images.

7. In a signal processing device for restoring onboard SAR images, RF transceiver for transmitting and receiving analog signals to and from a target; A navigation sensor unit that senses navigation data of the above signal processing device; A signal processing unit that converts an analog signal received by the above RF transceiver into a digital signal; Memory section; and Includes a processor unit; and The above memory unit It is configured to store the digital signal converted by the signal processing unit and the sensed navigation data, and The above signal processing unit The above converted digital signal is configured to process the SAR signal by mounting a high-speed conversion algorithm specialized for the conversion of the SAR image on a GPU device, Signal processing device for restoring onboard SAR images.

8. A method in which a satellite image 3D conversion device generates a 3D model based on an image of a satellite image using a deep learning model, A step in which a processor unit of the satellite image 3D conversion device generates a digital elevation model based on an image of the satellite image using the deep learning model; A point cloud generation step in which the processor unit generates a point cloud based on the generated digital elevation model; A removal step in which the processor unit removes at least one of the noise and outliers of the generated point cloud; The above processor unit generates a mesh based on a point cloud from which at least one of the noise and outliers has been removed; and The above processor unit generates a 3D model based on the generated mesh; comprising method.

9. In Paragraph 8, The above point cloud generation step; The above processor unit further includes a pixel coordinate extraction step for extracting the X, Y, and Z values ​​of each pixel of the digital elevation model. method.

10. In Paragraph 9, The pixel coordinate extraction step above; The above processor unit further includes a Z-coordinate scaling step for reflecting the actual altitude of the satellite image with respect to the extracted Z value. method.

11. In Paragraph 10, The above Z-coordinate scaling step; is The above processor unit further comprises the step of removing outliers from the Z value of the pixel or performing supplementation using interpolation with respect to missing values ​​by utilizing surrounding data. method.

12. In Paragraph 10, The above point cloud generation step; The above processor unit further comprises the step of integrating the extracted X, Y, and Z coordinate data to generate a reference point in 3D space. method.

13. In Paragraph 12, The above point cloud generation step; The above processor unit performs a search for neighbor points around the generated reference point; and The above processor unit further comprises the step of calculating a normal vector based on the generated reference point and the searched neighbor point. method.

14. In Paragraph 8, The above removal step; is The above processor unit further comprises the step of removing at least one point among points having a density below a threshold and points located at a position above a threshold from the expected position with respect to the generated point cloud. method.

15. A method for detecting algal blooms using an algal bloom detection device that detects algal bloom areas within a body of water using received satellite image data, A preprocessing step in which the processor unit of the above-mentioned algal bloom detection device preprocesses the received satellite image data; An extraction step in which the processor unit extracts first water body data from the preprocessed satellite image data; A segmentation step in which the processor unit performs deep learning-based segmentation based on the extracted first water body data to generate second water body data; A clustering generation step in which the processor unit generates a cluster based on the generated second body data; and A detection mask generation step comprising the above processor unit generating a detection mask of the green algae region based on the generated cluster; method.

16. In Paragraph 15, The above preprocessing step; is, A step in which the processor unit analyzes the received satellite image data to separate the R (red) channel, G (green) channel, and B (blue) channel; The above processor unit performs correction by adjusting the balance of at least one of the separated channel-specific brightness and color; and The above processor unit further comprises the step of integrating each channel on which the above correction has been performed. method.

17. In Paragraph 15, The above extraction step; is, The above processor unit further comprises the step of distinguishing a non-water area of ​​the received satellite image data and a water area including the first water area data by utilizing at least one segmentation technique among a color-based filtering technique, a brightness-based filtering technique, and a texture-based filtering technique. method.

18. In Paragraph 15, The above segmentation step; is, The above processor unit further comprises the step of distinguishing detailed regions within a water body for the extracted first water body data by applying at least one deep learning-based algorithm among U-Net, Fully Convolutional Network, and DeepLab. method.

19. In Paragraph 15, The above segmentation step; is, The above processor unit further comprises a prediction step of predicting the class of each pixel of image data containing the first water body data by utilizing a pre-trained deep learning model. method.

20. In Paragraph 19, The above prediction step; is, The above processor unit further comprises the step of predicting whether each pixel of image data containing the first water body data corresponds to an algal bloom area, a water area, or other area by utilizing a pre-trained deep learning model. method.

Images