Method for processing and analyzing data including hyperspectral image on basis of artificial general intelligence (AGI) and foundation model (FM)

Abstract

The present invention relates to a method for processing and analyzing data including a hyperspectral image on the basis of artificial general intelligence (AGI) and a foundation model (FM) and, more specifically, to a method for processing and analyzing data including a hyperspectral image on the basis of artificial general intelligence (AGI) and a foundation model (FM), the method comprising: receiving a request for a task to be performed on a hyperspectral image from a user; determining, using an LLM, whether the task needs to be processed by a first task processing module based on deep learning or a second task processing module based on a rule base; and when the hyperspectral image needs to be processed by the first task processing module based on deep learning, inputting feature information extracted by inputting the hyperspectral image to an encoder into a first sub-task module based on an artificial neural network to derive a task result for the task requested by the user to be performed on the hyperspectral image.

Description

Methods for working with and analyzing data including hyperspectral images based on AGI (Artificial General Intelligence) and FM (Fundation Model)

[0001] The present invention relates to a method for performing operations and analysis on data including hyperspectral images based on Artificial General Intelligence (AGI) and Foundation Model (FM), wherein the method receives a request from a user to perform an operation on a hyperspectral image, determines using an LLM whether the operation should be processed by a deep learning-based first operation processing module or a rule-based second operation processing module, and if the hyperspectral image is to be processed by the deep learning-based first operation processing module, inputs the hyperspectral image into an encoder to extract feature information, inputs the extracted feature information into an artificial neural network-based first detailed operation module, and derives the operation result for the operation the user intends to perform on the hyperspectral image.

[0002]

[0003] Hyperspectral imaging refers to images captured using extensive spectral data that includes visible light and near-infrared regions, unlike conventional images distinguished by the three primary colors of light. While humans perceive only three bands (RGB), hyperspectral imaging can contain information at various wavelengths that are imperceptible to the human eye, making it widely applicable in diverse technological fields such as agriculture, geology, medicine, food, environment, marine, and national defense.

[0004] As such, among the types of tasks that can be preprocessed or performed on hyperspectral images, there are tasks that can be performed by rule-based algorithms, such as selecting only pixels whose reflectance exceeds a specific value among wavelengths in a specific band or calculating similarity between pixels, and there are also tasks that must be performed using deep learning-based machine learning models including trained artificial neural networks, such as removing noise from parts presumed to be land, improving the resolution of parts corresponding to farmland, making foggy parts of rivers and forests clearer, or identifying oil spill areas.

[0005] As such, it is necessary to perform operations on hyperspectral images using different processing methods (rule-based or deep learning-based) depending on the user's request. However, conventionally, there has been no technology provided for a task processing module that performs operations on hyperspectral images based on deep learning and a task processing module that performs operations on hyperspectral images based on rule-based methods, which analyzes the user's request to perform operations on hyperspectral images and performs operations on hyperspectral images according to the user's request using either of the two task processing modules.

[0006]

[0007] Meanwhile, when using a task processing module that performs operations on hyperspectral images based on deep learning, feature information is extracted from the hyperspectral image, and detailed operations according to the user's request can be performed based on the extracted feature information.

[0008] Specifically, feature information can be extracted by inputting it into a trained artificial neural network-based encoder, and the performance of the encoder needs to be guaranteed so that the encoder semantically compresses the hyperspectral image to smoothly extract the feature information.

[0009] However, as mentioned above, hyperspectral images are data with a three-dimensional spectral structure containing data for multiple channels according to wavelength, and compared to conventional images, there is a problem in that it is difficult to obtain a large amount of data for training. In particular, it is even more difficult to obtain labeled data for hyperspectral images.

[0010] For this reason, there is a problem in training an artificial neural network-based encoder that extracts feature information from hyperspectral images. Consequently, there is a need to propose a method that can efficiently and robustly train the said encoder using a small number of hyperspectral images.

[0011]

[0012] The present invention aims to provide a method for performing operations and analysis on data including hyperspectral images based on Artificial General Intelligence (AGI) and Foundation Model (FM), wherein the method receives a request from a user to perform an operation on a hyperspectral image, determines using LLM whether the operation should be processed by a deep learning-based first operation processing module or a rule-based second operation processing module, and if the hyperspectral image is to be processed by the deep learning-based first operation processing module, inputs the hyperspectral image into an encoder to extract feature information, inputs the extracted feature information into an artificial neural network-based first detailed operation module, and derives the operation result for the operation the user intends to perform on the hyperspectral image.

[0013]

[0014] To solve the above problem, a method for performing a task on a hyperspectral image according to a user's task request comprises: a task request reception step of receiving a task request in which information to be performed on the hyperspectral image is entered in text form from a user terminal; a task attribute information derivation step of configuring a prompt including the task request and a phrase requesting judgment of necessary task attribute information according to the task request, and inputting the prompt into an LLM to derive task attribute information for the task request; and a task execution step of determining, based on the task attribute information, which module to process the hyperspectral image is a deep learning-based first task processing module or a rule-based second task processing module, and inputting the hyperspectral image into the determined first task processing module or second task processing module to derive a task result for the hyperspectral image. The method may include a result provision step of providing the above work result to a user terminal; wherein the work attribute information includes information on a major attribute regarding which module, either a deep learning-based first work processing module or a rule-based second work processing module, should process the hyperspectral image, and information on a detailed attribute of a work that is further distinguished from the major attribute; and the work execution step may include a feature information extraction step of inputting the hyperspectral image into an artificial neural network-based encoder included in the first work processing module to extract feature information when the hyperspectral image needs to be processed in the first work processing module; and a work result derivation step of inputting the feature information into an artificial neural network-based first detailed work module corresponding to the detailed attribute of the work attribute information included in the first work processing module to derive a work result for the hyperspectral image.

[0015] In one embodiment of the present invention, the second task processing module includes one or more second detailed task modules that perform a task for each of one or more detailed attributes using a rule-based algorithm, and the task execution step may include a step of inputting the hyperspectral image to the second detailed task module corresponding to the detailed attribute of the task attribute information to derive a task result when the hyperspectral image needs to be processed by the second task processing module.

[0016] In one embodiment of the present invention, the first detailed task module may include one or more of a learned machine learning inference model that super-resolves the hyperspectral image, a machine learning inference model including an artificial neural network that removes noise from the hyperspectral image, and a learned machine learning inference model that identifies and segments objects in the hyperspectral image.

[0017] In one embodiment of the present invention, the hyperspectral image includes information on the wavelength of light for each pixel according to a plurality of channels according to wavelength, and the encoder may include an artificial neural network trained to extract feature information for the hyperspectral image.

[0018] In one embodiment of the present invention, the encoder is trained by performing the following steps, wherein the following steps include: a masking patch area determination step for determining a masking patch area to be randomly masked for each of the plurality of channels of a hyperspectral image having a plurality of channels according to wavelength, which is the subject of training; a masking step for masking a masking patch area in each of the plurality of channels of the hyperspectral image and dividing it into a plurality of patches; an encoding step for deriving first feature information for each of the plurality of unmasked patches by inputting the divided plurality of patches, excluding the masked portion of the hyperspectral image, into an artificial neural network-based encoder; an integrated feature information generation step for generating integrated feature information by inserting a plurality of masking feature information for the masked patches into the plurality of first feature information; and a decoding step for generating a reconstructed hyperspectral image by inputting the integrated feature information into an artificial neural network-based decoder. and a learning step of training an encoder and a decoder to minimize the difference between the hyperspectral image and the reconstructed hyperspectral image; wherein the integrated feature information may include, for each of the plurality of first feature information and the plurality of masking feature information, information about a corresponding position in the hyperspectral image or information about the order between data.

[0019] In one embodiment of the present invention, the masking step may include fixing the positions of a plurality of masked patches in each of the plurality of channels of the hyperspectral image, and rearranging the positions of at least some of the unmasked patches by randomly mixing them.

[0020]

[0021] According to one embodiment of the present invention, a correction factor for the light reflectance in the environment in which the reference sample is photographed is determined based on the measured value of the reflectance of the reference sample when the reference sample is photographed under natural light or artificial light and the reference value of the reflectance known for the reference sample, thereby generating a correction map for correcting the reflectance that fluctuates due to external environments such as light scattering when a hyperspectral image is photographed.

[0022] According to one embodiment of the present invention, some of the correction maps may be generated based on experimental data obtained when a hyperspectral image of a reference sample is taken, while others may be generated by inputting arbitrary numerical values.

[0023] According to one embodiment of the present invention, a corrected hyperspectral image is input into an encoder and a decoder based on a learned artificial neural network to generate a reconstructed hyperspectral image, and the difference value between the corrected hyperspectral image and the reconstructed hyperspectral image can be determined as the difference value for the correction map that generated the corrected hyperspectral image.

[0024] According to one embodiment of the present invention, based on a correction coefficient set in each of a plurality of correction maps and a difference value determined for the corresponding correction map, the correction coefficient of the final correction map can be determined in a direction in which the difference value of the final correction map is minimized.

[0025] According to one embodiment of the present invention, by applying a final correction map to a hyperspectral image of an object, information regarding the actual reflectance of the object for each of multiple wavelengths can be derived.

[0026] According to one embodiment of the present invention, a foundation model including an encoder and a decoder can be trained based on a hyperspectral image in which patches randomly determined as masking patch regions are masked in each of a plurality of channels.

[0027] According to one embodiment of the present invention, a foundation model including an encoder and a decoder can be trained based on a hyperspectral image in which patches randomly determined as masking patch regions in each of a plurality of channels are masked and the positions of unmasked patches are rearranged.

[0028] According to one embodiment of the present invention, a foundation model including an encoder and a decoder can be trained based on a hyperspectral image in which patches randomly determined as masking patch regions in each of a plurality of channels are masked and noise is inserted into unmasked patches.

[0029] According to one embodiment of the present invention, a foundation model including an encoder and a decoder can be trained based on a hyperspectral image in which patches determined to be randomly masked as masking patch regions in each of a plurality of channels are masked, the positions of unmasked patches are rearranged, and noise is inserted.

[0030] According to one embodiment of the present invention, a foundation model including an encoder and a decoder can be trained multiple times while gradually increasing the number of patches corresponding to the masking patch region in the hyperspectral image to be trained.

[0031] According to one embodiment of the present invention, a foundation model including an encoder and a decoder can be trained multiple times while gradually increasing the number of patches rearranged in a hyperspectral image to be trained.

[0032] According to one embodiment of the present invention, a foundation model including an encoder and a decoder can be trained multiple times while gradually increasing the intensity of noise inserted in a hyperspectral image to be trained.

[0033] According to one embodiment of the present invention, a foundation model including an encoder and a decoder can be trained multiple times while gradually increasing the number of patches corresponding to the masking patch region in the hyperspectral image to be trained, the number of patches being rearranged, and the intensity of the noise.

[0034] According to one embodiment of the present invention, a foundation model including an encoder and a decoder can be trained multiple times based on a hyperspectral image in which patches at the same location in each of a plurality of channels of the hyperspectral image are masked and unmasked patches are rearranged to different locations in each of a plurality of channels.

[0035] According to one embodiment of the present invention, using an LLM, work attribute information including information on major attributes and detailed attributes regarding a task that a user intends to perform on a hyperspectral image can be derived.

[0036] According to one embodiment of the present invention, based on the main attributes of the task attribute information, it can be determined which module, either a deep learning-based first task processing module or a rule-based second task processing module, should process the task that the user intends to perform on the hyperspectral image.

[0037] According to one embodiment of the present invention, based on the detailed attributes of the task attribute information, it is possible to determine which module among the plurality of first detailed task processing modules of the first task processing module should process the task that the user intends to perform on the hyperspectral image.

[0038] According to one embodiment of the present invention, based on the detailed attributes of the task attribute information, it is possible to determine which module among the plurality of second detailed task processing modules of the second task processing module should process the task that the user intends to perform on the hyperspectral image.

[0039]

[0040] FIG. 1 illustrates a service server that performs a method for refining hyperspectral images using deep learning according to an embodiment of the present invention.

[0041] FIG. 2 illustrates the steps of a method for refining a hyperspectral image using deep learning according to an embodiment of the present invention.

[0042] FIG. 3 illustrates the reflectance of an object according to the scattering rate of the atmosphere when a hyperspectral image is taken according to one embodiment of the present invention.

[0043] Figure 4 illustrates a method for deriving the actual reflectance of an object in the prior art.

[0044] FIG. 5 illustrates a hyperspectral image according to one embodiment of the present invention.

[0045] FIG. 6 illustrates a method for generating a correction map according to an embodiment of the present invention.

[0046] FIG. 7 illustrates a corrected hyperspectral image generated by applying a correction map to a hyperspectral image according to one embodiment of the present invention.

[0047] FIG. 8 illustrates details regarding a corrected hyperspectral image, a reconstructed hyperspectral image, and a difference value according to an embodiment of the present invention.

[0048] FIG. 9 illustrates a method for generating a final correction map according to an embodiment of the present invention.

[0049] FIG. 10 illustrates a method for learning a foundation model for a hyperspectral image according to a preferred embodiment of the present invention.

[0050] FIG. 11 illustrates a method for training a foundation model according to one embodiment of the present invention.

[0051] FIG. 12 illustrates a method for training a foundation model while rearranging patches according to an embodiment of the present invention.

[0052] FIG. 13 illustrates a method for training a foundation model while adding noise according to an embodiment of the present invention.

[0053] FIG. 14 illustrates a method for training a foundation model so that the difference between a hyperspectral image and a reconstructed hyperspectral image is minimized according to one embodiment of the present invention.

[0054] FIG. 15 illustrates a method for training a foundation model while gradually increasing the size of a masking patch area according to one embodiment of the present invention.

[0055] FIG. 16 illustrates a method for training a foundation model while gradually increasing the number or ratio of rearranged patches according to one embodiment of the present invention.

[0056] FIG. 17 illustrates a method for training a foundation model while gradually increasing noise according to an embodiment of the present invention.

[0057] FIG. 18 illustrates patches that are rearranged by a plurality of channels of a hyperspectral image according to one embodiment of the present invention.

[0058] FIG. 19 illustrates a service server that performs a method for performing a task according to a user's task request on a hyperspectral image according to an embodiment of the present invention.

[0059] FIG. 20 illustrates details regarding a work request and work attribute information according to an embodiment of the present invention.

[0060] FIG. 21 illustrates details regarding a first work processing module according to an embodiment of the present invention.

[0061] FIG. 22 illustrates details regarding a second task processing module according to an embodiment of the present invention.

[0062] FIG. 23 illustrates examples of operations performed on a hyperspectral image according to one embodiment of the present invention.

[0063] FIG. 24 schematically illustrates the internal configuration of a computing device according to one embodiment of the present invention.

[0064]

[0065] Hereinafter, various embodiments and / or aspects are disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will also be recognized by those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the description is intended to include all such aspects and their equivalents.

[0066]

[0067] In addition, various aspects and features will be presented by a system that may include multiple devices, components and / or modules, etc. It should also be understood and recognized that various systems may include additional devices, components and / or modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in relation to the drawings.

[0068] Terms such as “embodiment,” “example,” “aspect,” “example,” etc. as used herein may not be interpreted as implying that any aspect or design described is superior or advantageous to other aspects or designs. Terms used below, such as “part,” “component,” “module,” “system,” “interface,” etc., generally refer to computer-related entities and may refer, for example, to hardware, a combination of hardware and software, or software.

[0069] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that the relevant feature and / or component is present, but not to exclude the presence or addition of one or more other features, components and / or groups thereof.

[0070] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0071] Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0072]

[0073] 1. Method for refining hyperspectral images using deep learning

[0074]

[0075] Before providing a detailed description of the present invention, “1. Method for refining hyperspectral images using deep learning” will be described. Specifically, the foundation model including the learned encoder and decoder described above in “1. Method for refining hyperspectral images using deep learning” can be generated or learned by “2. Method for generating a foundation model for hyperspectral images” described below.

[0076] That is, the foundation model can perform the process described in “1. Method for refining hyperspectral images using deep learning” after being trained through the process described in “2. Method for generating a foundation model for hyperspectral images”.

[0077]

[0078] FIG. 1 illustrates a service server (1) that performs a method for refining a hyperspectral image using deep learning according to an embodiment of the present invention.

[0079]

[0080] In FIG. 1, the encoder (100) and decoder (101) may be trained such that the difference between the training data input to the encoder (100); the output value when the training data is input to the encoder (100) to extract feature information and the extracted feature information is input to the decoder (101) is minimized.

[0081]

[0082] The service server (1) of the present invention can refine a hyperspectral image to generate a refined hyperspectral image. Specifically, a hyperspectral image may be an image containing information about the reflectance of an object when the object is photographed with a hyperspectral camera. Preferably, the hyperspectral image may include information about the reflectance of an object in which the reflectance of the object is distorted due to light scattering or noise.

[0083] Meanwhile, a refined hyperspectral image may be an image that contains information about the actual reflectance of an object without reflectance distortion.

[0084] For example, the service server (1) can apply a final correction map to the reflectance of an object photographed at 400 nm to 1000 nm to derive information about the actual reflectance of the object at 400 nm to 1000 nm.

[0085]

[0086] The service server (1) may include a foundation model (10) that includes an artificial neural network-based encoder (100) and a decoder (101). Specifically, the encoder (100) may include an artificial neural network that compresses input data to extract feature information. Additionally, the decoder (101) may include an artificial neural network that receives feature information output from the encoder (100) and restores it to be similar to the data input to the encoder (100).

[0087] Preferably, each of the encoder (100) and the decoder (101) can be trained such that the difference between the training data input to the encoder (100) and the output value when the feature information extracted by inputting the training data to the encoder (100) is input to the decoder (101) is minimized.

[0088]

[0089] In one embodiment of the present invention, the encoder (100) of the foundation model (10) may be a Vision Transformer (ViT). Specifically, the Vision Transformer-based foundation model (10) (or encoder (100)) can divide an input hyperspectral image into a plurality of patches of a fixed size and embed each of the plurality of patches to extract feature information.

[0090] In one embodiment of the present invention, a vision transformer-based encoder (100) receives a patch divided into a fixed size in each of the channels distinguished in a hyperspectral image, and feature information for each of the patches can be extracted.

[0091]

[0092] FIG. 2 illustrates the steps of a method for refining a hyperspectral image using deep learning according to an embodiment of the present invention.

[0093]

[0094] As illustrated in FIG. 2, a method for refining hyperspectral images using deep learning comprises: a correction step of generating multiple corrected hyperspectral images by applying multiple correction maps to hyperspectral images containing pixel-by-pixel unit information for each of the multiple channels according to wavelength; a feature information extraction step of inputting the multiple corrected hyperspectral images into an encoder (100) including an artificial neural network based on an artificial neural network to extract feature information for each of the input multiple corrected hyperspectral images; a restoration step of inputting the multiple feature information into an artificial neural network-based decoder (101) to generate a restored hyperspectral image such that the difference from the hyperspectral image that generated the feature information is minimized; a difference value derivation step of determining the difference value between the corrected hyperspectral image and the restored hyperspectral image as the difference value for the correction map used when generating the corrected hyperspectral image; and a final correction map generation step of generating a final correction map such that the difference value is minimized based on the difference value for each of the multiple correction maps. It may include a purification step of generating a purified hyperspectral image by applying the final correction map to the hyperspectral image.

[0095]

[0096] The service server (1) can first generate a final correction map for a hyperspectral image to be refined, and then apply the final correction map to the hyperspectral image to generate a refined hyperspectral image, and the steps are described in detail below.

[0097]

[0098] In step S1, the service server (1) can generate a plurality of correction maps. Specifically, a correction map is data for correcting the distortion of reflectance caused by light scattering, etc., when a hyperspectral image is captured, and may include a correction coefficient for how much to correct the reflectance in the hyperspectral image.

[0099] Preferably, the correction factor may include a correction factor for each of the multiple channels (wavelengths) distinguished in the hyperspectral image.

[0100]

[0101] In step S2, the service server (1) can generate a corrected hyperspectral image by applying each of the multiple correction maps to the hyperspectral image.

[0102] For example, a hyperspectral image contains information about reflectance, and when a correction map is applied to the hyperspectral image, the information about reflectance can be adjusted according to the correction coefficients of the correction map.

[0103]

[0104] In step S3, the service server (1) can input the corrected hyperspectral image into the encoder (100) to extract feature information. Specifically, each of the multiple corrected hyperspectral images can be input into the encoder (100) to extract feature information for each of the corrected hyperspectral images.

[0105] In one embodiment of the present invention, the feature information may be an embedding vector obtained by compressing a corrected hyperspectral image.

[0106]

[0107] In step S4, the service server (1) can input feature information for each corrected hyperspectral image into the decoder (101) to generate a reconstructed hyperspectral image.

[0108] Specifically, the decoder (101) may include an artificial neural network trained to receive feature information and generate an image similar to the corrected hyperspectral image that generated the feature information.

[0109]

[0110] In step S5, the service server (1) can derive a difference value between the corrected hyperspectral image and the reconstructed hyperspectral image. Specifically, the service server (1) can derive a difference value between the corrected hyperspectral image input to the encoder (100) and the reconstructed hyperspectral image output from the decoder (101).

[0111] In addition, the difference value can be determined as the difference value for the correction map used when generating the corresponding corrected hyperspectral image.

[0112]

[0113] In one embodiment of the present invention, the smaller the difference value, the more appropriately the correction map is set. Specifically, the correction coefficient of the correction map may be a coefficient applied to the reflectance of an object in a hyperspectral image to account for the fact that the reflectance of an object in a hyperspectral image is measured differently from the actual value due to reasons such as light scattering.

[0114] In other words, the better the correction coefficients of the correction map reflect factors that cause the object's reflectance to fluctuate, such as light scattering, the smaller the aforementioned difference value can be calculated. Conversely, the less well the correction coefficients of the correction map reflect factors that cause the object's reflectance to fluctuate, such as light scattering, the smaller the aforementioned difference value can be calculated.

[0115] In other words, the more appropriately a correction map with a correction factor is set, the more similar the 'corrected hyperspectral image generated using that correction map' and the 'reconstructed hyperspectral image generated from that corrected hyperspectral image' can be.

[0116]

[0117] In step S6, the service server (1) can generate a final correction map such that the difference value is minimized. Specifically, through step S5, the service server (1) can determine the difference value for each of the multiple correction maps, and based on the correction coefficient and the difference value for each of the multiple correction maps, can determine the correction coefficient of the final correction map in a direction such that the difference value of the final correction map is minimized.

[0118]

[0119] In step S7, the service server (1) can generate a refined hyperspectral image by applying a final correction map to the hyperspectral image.

[0120]

[0121] FIG. 3 illustrates the reflectance of an object according to the scattering rate of the atmosphere when a hyperspectral image is taken according to one embodiment of the present invention.

[0122]

[0123] As shown in Fig. 3(a), when there is an object with a light reflectance of 100% at a specific wavelength (e.g., 400 nm), the hyperspectral image taken of the object must show the light reflectance at that wavelength as 100%.

[0124]

[0125] However, as shown in Fig. 3(b), in a hyperspectral image actually taken of the object, the reflectance of light in the channel for a wavelength of 400 nm can be captured (measured) as 30%. This may be due to the phenomenon of light scattering by water vapor, aerosols, etc. in the atmosphere.

[0126] For example, the amount of light reflected from an object varies depending on light scattering, and accordingly, information regarding reflectance in hyperspectral images may be captured (measured) as higher or lower than the actual reflectance of the object.

[0127]

[0128] As such, the greater the factor distorting reflectance, such as light scattering, the greater the difference between the actual reflectance of the object and the reflectance of the object captured through hyperspectral imaging, depending on the environment (light scattering rate) at the time the object is photographed.

[0129]

[0130] In one embodiment of the present invention, the correction coefficients in the correction map may be coefficients for correcting between the 'actual reflectance of the object' and the 'reflectance when the object is photographed'.

[0131] For example, in FIGS. 3 (a) and (b), when the correction factor of the correction map in the channel for a wavelength of 400 nm is set to 3.33, when the reflectance of the object in the hyperspectral image is 30%, applying the correction factor 3.33 results in a reflectance of approximately 100%, which is similar to the actual reflectance of the object, which is 100%. In other words, this can be understood as the correction factor of the correction map appropriately reflecting the environment in which the object was captured.

[0132] On the other hand, in cases where the correction factor does not reflect the environment in which the object is photographed, for example, when the correction factor of the correction map is set to 1.1, applying the correction factor of 1.1 when the reflectance is 30% results in a reflectance of approximately 33%, which is significantly different from the actual reflectance of the object, which is 100%.

[0133]

[0134] As such, in order to accurately calculate the actual reflectance of an object, the correction coefficient of the final correction map needs to be set to accurately reflect factors such as the ratio at which the reflectance changes due to environmental elements in which the object was photographed.

[0135]

[0136] In one embodiment of the present invention, the actual reflectance of an object can be calculated based on the correction coefficient of a correction map and the measured value of the reflectance captured for the object.

[0137] For example, in the example described above, the actual reflectance of an object was calculated by multiplying the correction coefficient of the correction map by the measured reflectance value captured for the object; however, this is not limited to this, and the actual reflectance of an object can be calculated by applying a mathematical method to the correction coefficient of the correction map and the measured reflectance value captured for the object.

[0138]

[0139] Figure 4 illustrates a method for deriving the actual reflectance of an object in the prior art.

[0140]

[0141] Conventionally, in order to derive the actual reflectance of an object, it is necessary to have the measured values ​​of the reflectance of the reference sample and the object, as well as information regarding the actual reflectance of the reference sample known to the reference sample.

[0142] Specifically, (the actual reflectance of the object) can be determined based on (the measured reflectance of the object) and (the scattering rate of light).

[0143] The reflectance measurement of an object can be understood as information included in the hyperspectral image of the object.

[0144] The scattering rate of light may be information about how much the reflectance changes in the environment when an object is captured in a hyperspectral image, and as mentioned above, this may vary depending on the environment (scattering rate, etc.) when captured in a hyperspectral image.

[0145] In other words, light scattering can be understood as a correction value for the reflectance of the environment when an object is captured as a hyperspectral image.

[0146]

[0147] Meanwhile, conventionally, in order to obtain information regarding the scattering rate of light, it is necessary to obtain a hyperspectral image by photographing an object in the same environment in which information regarding the actual reflectance of a reference sample is known.

[0148] Specifically, for a reference sample for which information regarding the actual reflectance is known as (A) of FIG. 4, a hyperspectral image can be taken in the same environment as when the object was photographed to obtain a measurement of the reflectance of the reference sample, and consequently, the scattering rate of light in the environment in which the object or the reference sample was photographed can be derived based on (actual reflectance of the reference sample) and (measurement of the reflectance of the reference sample).

[0149] In this way, information on the light scattering rate is obtained in the past, and based on the light scattering rate and the reflectance measurement of the object as shown in (B) of FIG. 4, the actual reflectance of the object can finally be derived.

[0150]

[0151] The actual reflectance of an object or reference sample can be understood as the ground truth of the reflectance of the object or reference sample.

[0152]

[0153] As such, in the conventional method, information about the actual reflectance of a reference sample is required to obtain information about the actual reflectance of an object.

[0154] That is, the hyperspectral image is a multidimensional spectral structure containing information on reflectance for multiple wavelengths, and in order to obtain information on the actual reflectance of an object at all wavelengths, information on the actual reflectance of a reference sample at all wavelengths must be known as ground truth.

[0155] However, in reality, there is a problem in obtaining a reference sample for which information on reflectance at all wavelengths is known.

[0156]

[0157] In addition, in conventional methods, it is necessary to capture a hyperspectral image of a reference sample in the same environment in which a hyperspectral image of the object was captured.

[0158] For example, when an object is photographed in an environment where the light scattering rate is 30% and a measurement of the object's reflectance is obtained, it is also necessary to obtain a measurement of the reflectance of the reference sample by photographing a hyperspectral image of the reference sample in the same environment.

[0159] In other words, there is a problem in that information regarding the light scattering rate must be re-derived for each environment in which a hyperspectral image of an object is captured.

[0160]

[0161] The present invention improves upon such conventional problems and enables the derivation of information regarding the actual reflectance of an object in a manner that does not require the process of obtaining a measured value of the reflectance of a reference sample and the actual reflectance of the reference sample.

[0162]

[0163] Hereinafter, a method for obtaining information regarding the actual reflectance of an object in the present invention will be described. Specifically, information regarding the actual reflectance of an object can be understood as a refined hyperspectral image.

[0164]

[0165] FIG. 5 illustrates a hyperspectral image according to one embodiment of the present invention.

[0166]

[0167] As illustrated in FIG. 5(a), the hyperspectral image may be data of a multidimensional spectral structure containing information about the reflectance of an object in each of the plurality of channels corresponding to the wavelength.

[0168] Specifically, in a hyperspectral image, information regarding the reflectance of an object for a specific channel can be composed of pixel-by-pixel unit information in that channel.

[0169] Preferably, the hyperspectral image may be data of a three-dimensional structure having two spatial dimensions (x,y) and a wavelength-dependent channel (z).

[0170]

[0171] As illustrated in FIG. 5(b), the unit information stored in each of the multiple pixels in each of the multiple channels in a hyperspectral image may include information about the reflectance of an object in the corresponding pixel.

[0172] Specifically, the hyperspectral image may include information on reflectance (1 to 16) for each of the channels ch1 to 224.

[0173] In a preferred embodiment of the present invention, a hyperspectral image may be patched by channel units including two or more channels among a plurality of channels. For example, in a hyperspectral image including channels ch1 to 224, patches may be divided by channel units from ch1 to 56, channel units from ch57 to 112, channel units from ch113 to 168, and channel units from ch169 to 224.

[0174] As such, in the present invention, the hyperspectral image can be divided into patches of fixed size with respect to spatial information (x,y axes) and channels (z axis).

[0175] In other words, hyperspectral images can be patched into cube shapes.

[0176]

[0177]

[0178] FIG. 6 illustrates a method for generating a correction map according to an embodiment of the present invention.

[0179]

[0180] As illustrated in FIG. 6, the hyperspectral image is data of a multidimensional spectral structure composed of pixel-by-pixel unit information containing information on the reflectance of a captured object for each of a plurality of channels corresponding to a wavelength, and the correction map and the final correction map include correction coefficients for the reflectance of light for each of the plurality of channels in the hyperspectral image, and the corrected hyperspectral image may include information on the reflectance of an object when assuming that the reflectance of an object in the hyperspectral image is measured differently from the actual value according to the correction coefficients of the correction map.

[0181] In addition, the correction map can be generated for the environment in which the reference sample was photographed, based on the measured value of the reflectance measured for the reference sample when the reference sample was photographed under natural light or artificial light; and the reference value of the reflectance stored for the reference sample.

[0182] Alternatively, some of the multiple correction maps may be generated for the environment in which the reference sample was photographed based on the measured value of reflectance of the reference sample when the reference sample was photographed under natural light or artificial light; and the reference value of reflectance stored for the reference sample; and other parts of the multiple correction maps may be generated randomly.

[0183]

[0184] First, the service server (1) can generate and store multiple correction maps. Specifically, the correction map may include correction coefficients for how much the reflectance needs to be corrected in each of the multiple channels of the hyperspectral image.

[0185]

[0186] As illustrated in FIG. 6(a), a correction map can be generated for the environment in which the reference sample was photographed based on the measured value of the reflectance of the reference sample when the reference sample was photographed under natural light or artificial light and the reference value of the reflectance stored for the reference sample.

[0187] Here, the reference value of reflectance may be the actual reflectance of the reference sample, i.e., ground truth. In addition, the measured reflectance value of the reference sample may be information about the reflectance of the reference sample obtained by taking a hyperspectral image of the reference sample in a specific environment.

[0188]

[0189] For example, in a hyperspectral image of a reference sample taken under natural light, if the measured reflectance of the reference sample in the 400 nm wavelength channel is 17.3% and the known reference value (actual reflectance) of the reference sample in the 400 nm wavelength channel is 100%, the correction factor for the 400 nm wavelength channel can be determined to be 0.173.

[0190] In this way, correction coefficients (0.173, 0.152, 0.133, ...) of the correction map for each of the multiple channels distinguished in the hyperspectral image can be determined.

[0191]

[0192] Meanwhile, each of the multiple correction maps can be generated for the environment in which the reflectance measurement of the reference sample is obtained. In other words, the correction map may include information on how much the reflectance of the object changes in the environment in which the hyperspectral image of the reference sample is taken, or information on how much the reflectance needs to be corrected according to the said change.

[0193] For example, correction map 1 is generated based on a hyperspectral image of a reference sample taken under natural light with 10% humidity or a measurement of the reference sample, and may include information on how much the reflectance needs to be corrected under natural light with 10% humidity.

[0194] Alternatively, correction map 2 may include information on how much the reflectance should be corrected under 15% humidity artificial light based on a hyperspectral image of a reference sample taken under 15% humidity artificial light, or measurements of the reference sample.

[0195]

[0196] Each of the plurality of correction maps may include a plurality of correction coefficients corresponding to each of the plurality of channels distinguished in the hyperspectral image.

[0197]

[0198] In this way, the service server (1) can generate correction maps for each of the multiple environments based on hyperspectral images of reference samples taken in each of the multiple environments.

[0199]

[0200] In one embodiment of the present invention, a hyperspectral image can be refined even without a correction map corresponding to the same environment as when the hyperspectral image to be refined was taken.

[0201] Specifically, when refining a hyperspectral image taken under 99% humidity natural light, that is, when deriving information about the actual reflectance of an object using a hyperspectral image taken under 99% humidity natural light, the hyperspectral image can be refined (information about the actual reflectance of the object can be derived) without a correction map generated for a hyperspectral image taken of a reference sample under 99% humidity natural light.

[0202]

[0203] As illustrated in FIG. 6(b), some of the correction maps are generated for the environment in which the reference sample was photographed based on the measured value of reflectance of the reference sample when the reference sample was photographed under natural light or artificial light; and the reference value of reflectance stored for the reference sample; and some of the correction maps may be generated randomly.

[0204] Preferably, the correction coefficients included in the correction maps that are randomly generated among a plurality of correction maps can be determined randomly.

[0205]

[0206] For example, in Fig. 6(b), correction map 1, correction map 2, and correction map 3 may be correction maps generated by measuring the reflectance of a reference sample as described above in Fig. 6(a).

[0207] Meanwhile, in Fig. 6(b), the remaining correction maps are randomly generated correction maps, specifically, each of the multiple correction coefficients included in the correction map can be randomly set.

[0208] In one embodiment of the present invention, the correction factor may be any real number.

[0209]

[0210] In a preferred embodiment of the present invention, some of the plurality of correction maps are correction maps generated based on the measured reflectance of a reference sample and a reference value, and others may be correction maps generated randomly.

[0211]

[0212] FIG. 7 illustrates a corrected hyperspectral image generated by applying a correction map to a hyperspectral image according to one embodiment of the present invention.

[0213]

[0214] As illustrated in FIG. 7, the service server (1) can generate multiple corrected hyperspectral images by applying each of the multiple correction maps to the hyperspectral image.

[0215] Specifically, the hyperspectral image captures an object and includes information on reflectance at each of multiple wavelengths, but the reflectance may be distorted due to external environmental factors such as light scattering.

[0216] In addition, each of the multiple corrected hyperspectral images may be information that corrects for distortion of reflectance caused by external environments, such as light scattering, in the hyperspectral image.

[0217]

[0218] FIG. 8 illustrates details regarding a corrected hyperspectral image, a reconstructed hyperspectral image, and a difference value according to an embodiment of the present invention.

[0219]

[0220] As illustrated in FIG. 8(a), the service server (1) can input each of the plurality of corrected hyperspectral images into an encoder (100) that includes a learned artificial neural network to extract feature information for each of the plurality of corrected hyperspectral images.

[0221] Additionally, the service server (1) can input multiple feature information extracted from the encoder (100) into a decoder (101) that includes a learned artificial neural network to generate a reconstructed hyperspectral image for each of the multiple feature information (or the corrected hyperspectral image that generated the corresponding feature information).

[0222]

[0223] In one embodiment of the present invention, the encoder (100) can divide the corrected hyperspectral image into a plurality of patches in each channel and embed each patch to extract feature information.

[0224] Alternatively, the corrected hyperspectral image is divided into multiple patches in each channel, and the preprocessed information is input to the encoder (100), and the encoder (100) can embed each patch to extract feature information.

[0225]

[0226] In one embodiment of the present invention, the decoder (101) receives feature information output from the encoder (100) and can generate a new hyperspectral image, which is a reconstructed hyperspectral image. Specifically, the decoder (101) can generate a reconstructed hyperspectral image similar to the corrected hyperspectral image input to the encoder (100).

[0227] Alternatively, the service server can generate a reconstructed hyperspectral image using information output from the decoder (101).

[0228]

[0229] In this way, when a corrected hyperspectral image is input into a foundation model (10) including an encoder (100) and a decoder (101), a reconstructed hyperspectral image similar to the corrected hyperspectral image can be generated.

[0230]

[0231] Meanwhile, as described above, the encoder (100) and the decoder (101) can be trained such that the difference between the training data input to the encoder (100) and the output value when the feature information extracted by inputting the training data to the encoder (100) is input to the decoder (101) is minimized. In other words, the encoder (100) and the decoder (101) can be trained such that the difference between the 'corrected hyperspectral image input to the encoder (100)' and the 'reconstructed hyperspectral image output from the decoder (101)' is minimized (becomes similar).

[0232] In one embodiment of the present invention, a foundation model (10) including an encoder (100) and a decoder (101) can be trained with a HyperGlobal-450K dataset including 1,486 EO-1 HSI scene images of size 2,000 by 256 and 215 GF-5B scene images of size 2,000 by 2,000.

[0233] In one embodiment of the present invention, a foundation model (10) including an encoder (100) and a decoder (101) can be trained based on Spectral Earth data or data of various substances, including food, captured indoors.

[0234]

[0235] Meanwhile, the service server (1) can calculate the difference between the corrected hyperspectral image and the reconstructed hyperspectral image. Specifically, the service server (1) can calculate the difference between the ‘corrected hyperspectral image input to the encoder (100)’ and the ‘reconstructed hyperspectral image output from the decoder (101) in relation to the corrected hyperspectral image’.

[0236]

[0237] As illustrated in Fig. 8(b), the service server (1) can determine the difference value for the corrected hyperspectral image and the reconstructed hyperspectral image as the difference value for the correction map associated with the corrected hyperspectral image and the reconstructed hyperspectral image.

[0238] For example, the difference between 'corrected hyperspectral image 1 generated by applying correction map 1 to the hyperspectral image' and 'reconstructed hyperspectral image 1 generated based on corrected hyperspectral image 1' can be determined as the difference value for the correction map 1.

[0239] Likewise, the difference between 'corrected hyperspectral image 2 generated by applying correction map 2 to the hyperspectral image' and 'reconstructed hyperspectral image 2 generated based on corrected hyperspectral image 2' can be determined as the difference value for the correction map 2.

[0240]

[0241] In this way, the service server (1) can determine the difference value for each of the multiple correction maps.

[0242]

[0243] Meanwhile, in one embodiment of the present invention, the more accurately the correction coefficients of the correction map reflect the factors of fluctuating reflectance in the environment in which the hyperspectral image was taken, the smaller the difference value for the correction map can be calculated.

[0244] For example, the more accurately the correction coefficients of the correction map reflect the values ​​of reflectance fluctuations (distortion) caused by external factors in the environment where the hyperspectral image was captured, the more similar the reconstructed hyperspectral image is to the corrected hyperspectral image, and accordingly, the difference value of the correction map used to generate the corrected hyperspectral image can be determined to be small.

[0245] That is, in Fig. 8 (b), correction map 2, which has the smallest difference value of 0.0457, can be understood as reflecting the distortion of reflectance relatively most accurately, and correction map 4, which has the largest difference value of 0.0723, can be understood as not reflecting the distortion of reflectance relatively most accurately.

[0246]

[0247] FIG. 9 illustrates a method for generating a final correction map according to an embodiment of the present invention.

[0248]

[0249] As illustrated in FIG. 9, the final correction map generation step can generate a final correction map by deriving the correction coefficient of the final correction map such that the difference value of the final correction map is minimized, based on the correction coefficient included in each of the plurality of correction maps and the difference value calculated for the corresponding correction map.

[0250] In addition, the hyperspectral image is data in which information regarding reflectance is distorted due to external environmental factors including atmospheric scattering and noise when the object is photographed under natural or artificial light, and the refined hyperspectral image is data in which information regarding reflectance is not distorted due to external environmental factors including atmospheric scattering and noise when the object is photographed under natural or artificial light, and may include information regarding the actual reflectance of the object.

[0251]

[0252] As illustrated in FIG. 9 (a), the service server (1) can determine each of the multiple correction coefficients of the final correction map based on the difference value for each of the multiple correction maps so that the difference value of the final correction map is minimized.

[0253] In one embodiment of the present invention, the correction coefficient of a correction map for each of a plurality of channels and the difference value of the corresponding correction map are mapped to spatial coordinates, and the coordinate with the smallest difference value in the spatial coordinates is inferred, and the correction coefficient at that coordinate can be determined as the correction coefficient for the final correction map.

[0254] For example, in FIG. 9(a), the correction coefficient of the correction map for a specific channel in the two-dimensional coordinate system is set as the coordinate with respect to the first axis, and the difference value is set as the coordinate with respect to the second axis, so that each of the multiple correction maps can be mapped to the two-dimensional coordinate system.

[0255] That is, in Fig. 9 (a), the coordinates at the outermost edge are correction maps with a relatively large difference value of about 0.8, and the coordinates at the inner edge are correction maps with a relatively small difference value of about 0.5.

[0256] The service server (1) can determine the coordinate with the smallest difference value in spatial coordinates, that is, the coordinate where the difference value is predicted to converge to about 0, and calculate the correction coefficient at the corresponding coordinate and determine it as the correction coefficient of the final correction map.

[0257]

[0258] In one embodiment of the present invention, the service server (1) can derive correction coefficients of the final correction map using a particle swarm optimization algorithm (PSO).

[0259]

[0260] In one embodiment of the present invention, the correction coefficients of the final correction map are generated based on a plurality of correction maps, thereby enabling them to be generated in a way that more accurately reflects the distortion regarding reflectance.

[0261] Preferably, some of the multiple correction maps are generated based on experimental data regarding the reflectance measurements of a reference sample, while others are generated using arbitrary values, thereby enabling the generation of a final correction map in situations where experimental data is insufficient.

[0262] In addition, by using correction maps generated with arbitrary values, the particle swarm optimization algorithm can generate the final correction map while handling a wider variety of cases.

[0263] In addition, by using correction maps generated with arbitrary values, the influence of outliers in the experimental data can be minimized, and distortion of the final correction map can be prevented.

[0264]

[0265] As illustrated in FIG. 9(b), the service server (1) can derive channel-specific correction coefficients (0.138, 0.156, 0.178, ...) such that the difference value of the final correction map is minimized to about 0.0301, based on the channel-specific correction coefficients and difference values ​​for each of correction maps 1 to N.

[0266] In other words, the difference value of the final correction map may be smaller than the difference value of each of the multiple correction maps stored in the service server (1).

[0267]

[0268] As such, according to one embodiment of the present invention, correction coefficients of a final correction map for refining a hyperspectral image are determined to generate a final correction map, and a refined hyperspectral image can be generated by applying the final correction map to the hyperspectral image.

[0269]

[0270] In one embodiment of the present invention, the service server (1) can perform the above-described process for each of the different hyperspectral images to generate a final correction map and a refined hyperspectral image for each of the different hyperspectral images.

[0271] In this way, compared to the conventional method which requires hyperspectral images of reference samples taken in each environment in which different hyperspectral images are taken to obtain refined hyperspectral images for different hyperspectral images, according to one embodiment of the present invention, only the final correction map for the corresponding hyperspectral image is regenerated using a learned encoder (100) and decoder (101) without hyperspectral images of reference samples taken in different environments, thereby enabling efficient and convenient generation or correction of refined hyperspectral images.

[0272]

[0273] 2. Generation and Training Method of Foundation Models for Hyperspectral Images

[0274]

[0275] Below, a method for generating or training a foundation model (10) including the artificial neural network-based encoder (100) and decoder (101) described above will be explained.

[0276]

[0277] FIG. 10 illustrates a method of learning a foundation model (10) for a hyperspectral image according to a preferred embodiment of the present invention.

[0278]

[0279] As illustrated in FIG. 10, the hyperspectral image includes information on the reflectance of light per pixel for each of the multiple channels according to wavelength, and the foundation model (10) may include a Vision Transformer.

[0280]

[0281] Specifically, the service server (1) can train the foundation model (10) for the hyperspectral image by performing preprocessing such as masking the hyperspectral image to be trained, rearranging patches, and adding noise, and can train the encoder (100) and decoder (101) of the foundation using the preprocessed hyperspectral image.

[0282]

[0283] Preferably, the foundation model (10) can generate a ‘reconstructed hyperspectral image’ similar to the ‘hyperspectral image that is the subject of learning before preprocessing’ based on the input data.

[0284] In other words, the encoder (100) of the foundation model (10) is input with data in which the hyperspectral image is masked, the patches are rearranged, and noise is added, and the decoder (101) of the foundation model (10) can be trained to generate a reconstructed hyperspectral image similar to the hyperspectral image before masking, patch rearrangement, and noise addition.

[0285]

[0286] FIG. 11 illustrates a method for training a foundation model (10) according to one embodiment of the present invention.

[0287]

[0288] A method for learning a foundation model (10) for a hyperspectral image, comprising an artificial neural network-based encoder (100) and a decoder (101) as illustrated in FIG. 11, comprising: a masking patch area determination step for determining a masking patch area to be randomly masked for each of the plurality of channels for a hyperspectral image having a plurality of channels according to wavelength to be learned; a masking step for masking a masking patch area in each of the plurality of channels of the hyperspectral image and dividing it into a plurality of patches; an encoding step for inputting the plurality of divided patches, excluding the masked part of the hyperspectral image, into the artificial neural network-based encoder (100) to derive feature information for each of the plurality of unmasked patches; and an integrated feature information generation step for generating integrated feature information by inserting a plurality of masking feature information for the masked patches into the plurality of feature information. The method includes a decoding step of inputting the above integrated feature information into an artificial neural network-based decoder (101) to generate a reconstructed hyperspectral image; and a learning step of training an encoder (100) and a decoder (101) to minimize the difference between the hyperspectral image and the reconstructed hyperspectral image; wherein the above integrated feature information may include, for each of the plurality of feature information and the plurality of masking feature information, information about a corresponding position in the hyperspectral image or information about the order between data.

[0289]

[0290] For convenience of explanation, the entity training the foundation model (10) will be referred to as the service server (1) in the following description. That is, in one embodiment of the present invention, the service server (1) can train the foundation model (10) on its own.

[0291] Alternatively, in one embodiment of the present invention, the service server (1) may receive a foundation model (10) that has been learned from an external source.

[0292]

[0293] A hyperspectral image is data of a three-dimensional spectral structure containing data (images) for each of a plurality of channels according to wavelength, and the following processes ① to ⑧ can be performed on the data (images) for each of the plurality of channels.

[0294]

[0295] ① The service server (1) determines a masking patch area to be randomly masked in the hyperspectral image to be learned, and can mask the area corresponding to the masking patch area. Specifically, the masking patch area may be a local area of ​​part to be masked in the image at each of the multiple channels.

[0296] Preferably, in each of the plurality of channels according to the wavelength of the hyperspectral image, one or more of the plurality of patches dividing the image can be determined as masking patch regions. That is, the masking patch regions can be determined on a patch basis.

[0297] For example, in FIG. 11, the image in a specific channel of the hyperspectral image can be divided into 1 to 16 patches, and the 2nd, 5th, 6th, 7th, ..., 15th patches among these can be determined as masking patch regions.

[0298]

[0299] Or, in another embodiment of the present invention, the masking patch area may be determined in pixel units.

[0300]

[0301] In one embodiment of the present invention, in each of the plurality of channels of a hyperspectral image, any number of patches at any position may be determined as masking patch areas.

[0302] Preferably, the masking patch area can be determined at the same location (area) in each of the multiple channels.

[0303]

[0304] ② The service server (1) can divide the hyperspectral image into multiple patches for each of the multiple channels. As described above, the masking patch area can be determined on a patch basis, and some of the divided multiple patches may be patches that are masked because they correspond to the masking patch area, and the remaining parts may be patches that are not masked because they do not correspond to the masking patch area.

[0305] For example, when the image in a specific channel of the hyperspectral image in Fig. 11 is divided into patches, the 2nd, 5th, 6th, 7th, ..., 15th patches may be masked, and the 1st, 3rd, 4th, 8th, and 16th patches may not be masked.

[0306]

[0307] ③ The service server (1) can input the plurality of divided patches, excluding the masked portions among the plurality of divided patches in the hyperspectral image, into an artificial neural network-based encoder (100). Specifically, the service server (1) can input the patches that are not masked because they do not correspond to the masked patch area among the plurality of divided patches in the hyperspectral image into the encoder (100).

[0308] For example, in a specific channel of the hyperspectral image in FIG. 11, unmasked 1st, 3rd, 4th, 8th, and 16th patches can be input to the encoder (100).

[0309]

[0310] ④ Feature information for each of the multiple unmasked patches can be extracted in the encoder (100). Specifically, the encoder (100) may include an artificial neural network trained to extract feature information by compressing the information inherent in each of the multiple patches.

[0311] In one embodiment of the present invention, the feature information may be an embedding vector.

[0312] For example, in FIG. 11, the encoder (100) can extract feature information for each of the unmasked 1st, 3rd, 4th, 8th, and 16th patches.

[0313]

[0314] In one embodiment of the present invention, the service server (1) can store information regarding which patch among the plurality of patches distinguished in the hyperspectral image each of the plurality of feature information extracted from the encoder (100) is generated for.

[0315] For example, the service server (1) can store information that feature information for each of the 1st, 3rd, 4th, 8th, and 16th patches extracted from the encoder (100) corresponds to each of the 1st, 3rd, 4th, 8th, and 16th patches in the hyperspectral image.

[0316]

[0317] ⑤ The service server (1) can generate integrated feature information by inserting multiple masking feature information for a patch masked in multiple feature information extracted from the encoder (100).

[0318] Specifically, the service server (1) can generate integrated feature information by inserting masking feature information between feature information while considering the position or order of the patches that generated each of the multiple feature information in the hyperspectral image.

[0319] Preferably, the service server (1) can generate integrated feature information by placing feature information or masking feature information generated for a corresponding patch at a location or order corresponding to each of a plurality of patches of a hyperspectral image.

[0320] More preferably, the service server (1) can generate integrated feature information by placing feature information generated for a patch at a position or order corresponding to each of a plurality of patches of a hyperspectral image when the patch is an unmasked patch, and placing masking feature information when the patch is a masked patch.

[0321]

[0322] For example, in FIG. 11, feature information generated for the first patch is placed at a position or order corresponding to the first unmasked patch of the hyperspectral image, masking feature information is placed at a position or order corresponding to the second masked patch, and feature information generated for the third patch is placed at a position or order corresponding to the third unmasked patch, thereby generating integrated feature information by placing feature information or masking feature information to correspond to each position or order of the first to 16th patches of the hyperspectral image.

[0323]

[0324] In this way, the integrated feature information may include information about which location each of the multiple feature information and multiple masking feature information corresponds to in the hyperspectral image, or information about the order in which the data are arranged.

[0325]

[0326] In one embodiment of the present invention, the masking feature information may be feature information extracted by inputting a masked patch into an encoder (100). For example, the feature information extracted by inputting any one patch corresponding to the masking patch area into the encoder (100) may be determined as the masking feature information.

[0327] Or, in another embodiment of the present invention, the masking feature information may be data in which the data value is empty.

[0328]

[0329] ⑥ The service server (1) can input the generated integrated feature information into an artificial neural network-based decoder (101). Specifically, the decoder (101) may include an artificial neural network trained to generate a reconstructed hyperspectral image similar to the hyperspectral image based on the input integrated feature information.

[0330]

[0331] ⑦, ⑧ The decoder (101) can generate a reconstructed hyperspectral image. Alternatively, the decoder (101) can generate data for generating a reconstructed hyperspectral image, and the service server (1) can generate a reconstructed hyperspectral image based on the information output from the decoder (101).

[0332]

[0333] In one embodiment of the present invention, the process illustrated by ⑦ is performed in a hidden layer inside the decoder (101), and the decoder (101) can output a reconstructed hyperspectral image.

[0334]

[0335] The decoder (101) can generate a reconstructed hyperspectral image similar to the hyperspectral image based on feature information for some unmasked patches, without receiving feature information for each of the multiple patches divided in the hyperspectral image.

[0336]

[0337] Meanwhile, the encoder (100) and decoder (101) can be trained to minimize the difference between the hyperspectral image and the reconstructed hyperspectral image. Specifically, the better the performance of the encoder (100) and decoder (101), the more similar the reconstructed hyperspectral image can be generated to the hyperspectral image.

[0338]

[0339] FIG. 12 illustrates a method of training a foundation model (10) while rearranging patches according to an embodiment of the present invention. Specifically, the training method described in FIG. 12 may be a method of training a foundation model (10) with a higher difficulty level than the training method of FIG. 11.

[0340]

[0341] As illustrated in FIG. 12, the masking step may include fixing the positions of the masked patches in each of the multiple channels of the hyperspectral image, and rearranging the positions of at least some of the unmasked patches by randomly mixing them.

[0342]

[0343] Since the process of ① to ② in Fig. 12 is substantially the same as the process of ① to ② described in Fig. 11, the redundant explanation will be omitted.

[0344]

[0345] ③ The service server (1) can randomly mix and rearrange the positions of at least some of the unmasked patches in each of the multiple channels of the hyperspectral image.

[0346] Specifically, the service server (1) can fix the positions of the patches corresponding to the masking patch area and rearrange the positions of the unmasked patches randomly among themselves.

[0347] For example, in FIG. 12, the service server (1) can fix the positions of the 2nd, 5th, 6th, 7th, ..., 15th patches that are masked corresponding to the masking patch area in the hyperspectral image, and randomly mix the positions of the 1st, 3rd, 4th, 8th, and 16th patches that are not masked.

[0348] That is, the service server (1) can rearrange the patches by randomly shuffling them in such a way that the 1st patch is moved to the position of the 3rd patch, the 3rd patch to the position of the 4th patch, and the 4th patch to the position of the 1st patch.

[0349]

[0350] In one embodiment of the present invention, the number of patches whose positions are rearranged may be proportional to the learning difficulty.

[0351]

[0352] ④ The service server (1) can input a plurality of patches, among the plurality of patches divided in the hyperspectral image, into an artificial neural network-based encoder (100), wherein the positions are randomly rearranged excluding the masked portion.

[0353]

[0354] ⑤ In the encoder (100), feature information for each of the multiple unmasked patches can be extracted.

[0355] For example, in FIG. 12, the encoder (100) can extract feature information for each of the 1st, 3rd, 4th, 8th, and 16th patches whose positions have been rearranged.

[0356] At this time, the feature information generated for the 1st patch before rearrangement is considered to be the feature information generated for the 3rd patch, the feature information generated for the 3rd patch before rearrangement is considered to be the feature information generated for the 4th patch, and the feature information generated for the 4th patch before rearrangement can be considered to be the feature information generated for the 1st patch.

[0357]

[0358] ⑥ The service server (1) can generate integrated feature information by inserting multiple masking feature information for a patch masked in multiple feature information extracted from the encoder (100).

[0359] For example, at the location or order corresponding to the unmasked first patch of the hyperspectral image in Fig. 12, feature information generated for the fourth patch, which is considered to have been generated for the first patch but was actually generated for the repositioned fourth patch, can be placed.

[0360] In addition, masking feature information can be placed at a position or order corresponding to the second masked patch.

[0361] In addition, feature information that is considered to have been generated for the third patch but was actually generated for the first patch prior to relocation can be placed at a location or order corresponding to the unmasked third patch.

[0362] In addition, feature information that is considered to have been generated for the fourth patch but was actually generated for the third patch prior to relocation can be placed at a location or order corresponding to the unmasked fourth patch.

[0363]

[0364] In this way, the service server (1) can generate integrated feature information by inserting masking feature information between the feature information generated for the patches whose positions have been rearranged.

[0365]

[0366] ⑦, ⑧, ⑨ Service server (1) inputs integrated feature information generated based on feature information generated for patches that are not masked and whose positions have been rearranged into a decoder (101), and the decoder (101) can generate a reconstructed hyperspectral image similar to a hyperspectral image, and since this is substantially the same as the process of ⑥ to ⑧ described in FIG. 11, a redundant explanation is omitted.

[0367]

[0368] That is, the decoder (101) can generate a reconstructed hyperspectral image similar to the hyperspectral image based on feature information generated in a state where some patches in the hyperspectral image are masked and some unmasked patches are rearranged with their positions mixed.

[0369]

[0370] In this way, according to one embodiment of the present invention, by using a hyperspectral image in which not only a part is masked but also the position of the unmasked part is rearranged, and by training an encoder (100) and a decoder (101) to generate a restored image similar to a hyperspectral image in which the masking process and the positions of the patches are not rearranged, the foundation model (10) can be trained with a higher difficulty level than the method of training with a hyperspectral image that is only masked as described in FIG. 11.

[0371]

[0372] FIG. 13 illustrates a method of training a foundation model (10) while adding noise according to one embodiment of the present invention.

[0373]

[0374] As illustrated in FIG. 13, the masking step may mask the data of the patch corresponding to the masking patch area in each of the plurality of channels of the hyperspectral image, and insert noise into the data of at least some of the patches that do not correspond to the masking patch area.

[0375]

[0376] Since the process of ① to ② in Fig. 13 is substantially the same as the process of ① to ② described in Fig. 11, a redundant explanation will be omitted.

[0377]

[0378] ③ The service server (1) can insert noise into the data of at least some of the unmasked patches in each of the multiple channels of the hyperspectral image.

[0379] Specifically, the service server (1) can mask the patches corresponding to the masking patch area and insert noise into the data of the unmasked patches.

[0380] For example, each of the multiple patches may contain information about the reflectance in the corresponding channel, and the service server (1) may change the reflectance stored in the unmasked patch to any value.

[0381]

[0382] In one embodiment of the present invention, the intensity of noise inserted into each of the plurality of patches and the number of patches into which noise is inserted may be proportional to the learning difficulty.

[0383] In one embodiment of the present invention, the noise may include one or more of Gaussian noise, speckle noise, and Poisson noise.

[0384]

[0385] ④ The service server (1) can input a plurality of patches with noise inserted, excluding the masked portion among the plurality of patches divided in the hyperspectral image, into an artificial neural network-based encoder (100).

[0386]

[0387] ⑤ In the encoder (100), feature information for each of the multiple patches with noise inserted without masking can be extracted.

[0388] For example, in FIG. 13, the encoder (100) can extract feature information for each of the 1st, 3rd, 4th, 8th, and 16th patches with inserted noise.

[0389]

[0390] ⑥ The service server (1) can generate integrated feature information by inserting multiple masking feature information for a patch masked in multiple feature information extracted from the encoder (100).

[0391]

[0392] ⑦, ⑧, ⑨ The service server (1) inputs integrated feature information, including feature information generated for a patch with noise inserted, into the decoder (101), and the decoder (101) can generate a reconstructed hyperspectral image similar to the hyperspectral image, and since this is substantially the same as the process of ⑥ to ⑧ described in FIG. 11, a redundant explanation will be omitted.

[0393]

[0394] That is, the decoder (101) can generate a reconstructed hyperspectral image similar to the hyperspectral image based on feature information generated in a state where some patches in the hyperspectral image are masked and some unmasked patches have noise inserted.

[0395]

[0396] In this way, according to one embodiment of the present invention, by using a hyperspectral image in which not only is part of it masked but noise from the unmasked part is also inserted, and by training an encoder (100) and a decoder (101) to generate a restored image similar to a hyperspectral image without masking and noise inserted, the foundation model (10) can be trained with a higher difficulty level than the method of training with a hyperspectral image that is only masked as described in FIG. 11.

[0397]

[0398] FIG. 14 illustrates a method for training a foundation model (10) so that the difference between a hyperspectral image and a reconstructed hyperspectral image is minimized according to one embodiment of the present invention.

[0399]

[0400] The hyperspectral image (to be learned) may be the original data that is not masked, has not rearranged the positions between patches, and has not had noise inserted.

[0401] The reconstructed hyperspectral image may be reconstructed data in which the foundation model (10) is reconstructed similarly to the hyperspectral image based on data after masking, rearrangement of positions between patches, or insertion of noise.

[0402]

[0403] Meanwhile, the encoder (100) and decoder (101) of the foundation model (10) can be trained so that the difference between the hyperspectral image and the reconstructed hyperspectral image is minimized. Specifically, the smaller the difference between the hyperspectral image and the reconstructed hyperspectral image, that is, the more similar the reconstructed hyperspectral image is to the hyperspectral image, the better the performance of the foundation model (10) can be understood.

[0404] In other words, the above difference value can be used as a performance indicator for the above foundation model (10).

[0405]

[0406] FIG. 15 illustrates a method for training a foundation model (10) while gradually increasing the size of a masking patch area according to one embodiment of the present invention.

[0407]

[0408] A method for learning a foundation model (10) for a hyperspectral image, comprising an artificial neural network-based encoder (100) and a decoder (101) as illustrated in FIG. 15, comprising: a masking patch area determination step for determining a masking patch area to be randomly masked for each of the plurality of channels for a hyperspectral image having a plurality of channels according to wavelength to be learned; a masking step for masking a masking patch area in each of the plurality of channels of the hyperspectral image and dividing it into a plurality of patches; an encoding step for inputting the plurality of divided patches, excluding the masked part of the hyperspectral image, into the artificial neural network-based encoder (100) to derive feature information for each of the plurality of unmasked patches; and an integrated feature information generation step for generating integrated feature information by inserting a plurality of masking feature information for the masked patches into the plurality of feature information. A decoding step of inputting the above integrated feature information into an artificial neural network-based decoder (101) to generate a first reconstructed hyperspectral image; a learning step of training an encoder (100) and a decoder (101) so as to minimize the difference between the hyperspectral image and the first reconstructed hyperspectral image; a first masking patch area re-determination step of resetting a greater number of patches as masking patch areas than the masking patch areas determined in the masking patch area determination step for the hyperspectral image; a first re-masking step of masking the re-set masking patch areas in the hyperspectral image and dividing them into a plurality of patches; a first re-encoding step of inputting the plurality of divided patches into the first encoder (100), excluding the parts masked according to the re-set masking patch areas in the hyperspectral image, to derive feature information for each of the plurality of unmasked patches; A first integrated feature information regeneration step for generating integrated feature information by inserting a plurality of masking feature information for a patch masked in a plurality of the above feature information; a first re-decoding step for generating a second reconstructed hyperspectral image by inputting the integrated feature information into a decoder (101); and a retraining step for retraining the encoder (100) and the decoder (101) so as to minimize the difference between the hyperspectral image and the second reconstructed hyperspectral image;It includes, wherein the integrated feature information, each of the plurality of feature information and the plurality of masking feature information, may include information regarding a corresponding position in the hyperspectral image or information regarding the order between data.;

[0409]

[0410] In one embodiment of the present invention, the foundation model (10) can be trained while gradually increasing the learning difficulty. Specifically, the foundation model (10) can be trained while gradually increasing the size of the masking patch area in the hyperspectral image to be trained, that is, while gradually masking a wider area.

[0411]

[0412] Specifically, in each of steps 1 to 3 illustrated in FIG. 15, the foundation model (10) can be learned through the process described in FIG. 11.

[0413] However, the size or width of the masking patch area in the hyperspectral image being trained in each of steps 1 to 3 may differ.

[0414] Specifically, in step 1, the foundation model (10) is trained based on a hyperspectral image with 4 patches masked in the hyperspectral image, in step 2, the foundation model (10) is trained based on a hyperspectral image with 6 patches masked in the hyperspectral image, and in step 3, the foundation model (10) can be trained based on a hyperspectral image with 11 patches masked in the hyperspectral image.

[0415]

[0416] In this way, the first step may be a masking patch area determination step; a masking step; an encoding step; an integrated feature information generation step; a decoding step; and a learning step in the learning method of the foundation model (10), and the second and third steps may each be a first masking patch area re-determination step; a first re-masking step; a first re-encoding step; a first integrated feature information re-generation step; a first re-decoding step; and a re-learning step.

[0417] In addition, the number of patches determined as masking patch regions in the hyperspectral image being learned in the first masking patch region recombination step may be greater than the number of patches determined as masking patch regions in the hyperspectral image being learned in the masking patch region determination step.

[0418]

[0419] *

[0420] In addition, the steps up to the first masking patch area recombination step; the first remasking step; the first re-encoding step; the first integrated feature information regeneration step; the first re-decoding step; and the re-learning step may be repeated two or more times.

[0421] That is, the number of patches determined as masking patch regions in the hyperspectral image being learned in the first masking patch region recrystallization step performed a second time may be greater than the number of patches determined as masking patch regions in the hyperspectral image being learned in the first masking patch region recrystallization step performed a first time.

[0422]

[0423] In this way, the foundation model (10) can be trained by curriculum learning, which starts at a low difficulty level and gradually progresses to a higher difficulty level, so that it is trained efficiently and stably and reduces overfitting.

[0424]

[0425] FIG. 16 illustrates a method for training a foundation model (10) while gradually increasing the number or ratio of rearranged patches according to one embodiment of the present invention.

[0426]

[0427] A method for learning a foundation model (10) for a hyperspectral image, comprising an artificial neural network-based encoder (100) and a decoder (101) as illustrated in FIG. 16, comprising: a masking patch area determination step for determining a masking patch area to be randomly masked for each of the plurality of channels according to wavelength for a hyperspectral image to be learned; a masking step for masking a masking patch area in each of the plurality of channels of the hyperspectral image, rearranging at least some of the unmasked patches by randomly shuffling their positions, and dividing them into a plurality of patches; an encoding step for inputting the divided plurality of patches, excluding the masked part of the hyperspectral image, into the artificial neural network-based encoder (100) to derive feature information for each of the unmasked plurality of patches; and an integrated feature information generation step for generating integrated feature information by inserting a plurality of masking feature information for the masked patches into the plurality of feature information. A decoding step of generating a first reconstructed hyperspectral image by inputting the above integrated feature information into an artificial neural network-based decoder (101); a learning step of training an encoder (100) and a decoder (101) so as to minimize the difference between the hyperspectral image and the first reconstructed hyperspectral image; a second masking patch area recombination step of determining a masking patch area to be randomly masked for each of the multiple channels of a hyperspectral image having multiple channels according to wavelength that is the subject of learning; a second re-masking step of masking the masking patch area in each of the multiple channels of the hyperspectral image, randomly mixing and rearranging a number of patches greater than the number of patches rearranged in the masking step, and dividing into multiple patches; a second re-encoding step of inputting the divided multiple patches into an artificial neural network-based encoder (100), excluding the masked part of the hyperspectral image, to derive feature information for each of the multiple patches that are not masked; A second integrated feature information regeneration step for generating integrated feature information by inserting a plurality of masking feature information for a patch masked in a plurality of the above feature information;The method comprises: a second re-decoding step of inputting the above integrated feature information into an artificial neural network-based decoder (101) to generate a second reconstructed hyperspectral image; and a second re-learning step of training the encoder (100) and the decoder (101) so as to minimize the difference between the hyperspectral image and the second reconstructed hyperspectral image; wherein the above integrated feature information may include, for each of the plurality of feature information and the plurality of masking feature information, information regarding a corresponding position in the hyperspectral image or information regarding the order between data.

[0428]

[0429] In one embodiment of the present invention, the foundation model (10) can be trained while gradually increasing the learning difficulty. Specifically, the foundation model (10) can be trained while gradually increasing the number of patches rearranged in the hyperspectral image being trained, that is, while gradually rearranging a larger number of patches.

[0430]

[0431] Specifically, in each of steps 1 to 3 shown in FIG. 16, the foundation model (10) can be learned through the process described in FIG. 12.

[0432] However, among the multiple unmasked patches in the hyperspectral image being trained in each of steps 1 to 3, the number of rearranged patches may differ.

[0433] Specifically, in FIG. 16, the hyperspectral image includes 25 patches for each of the multiple channels, of which 5 patches marked in white are masked and 20 shaded patches may not be masked.

[0434] In step 1, the foundation model (10) can be trained based on randomly rearranged data of the positions of 5 patches (labeled 1 to 5) out of 20 unmasked patches.

[0435] In step 2, the foundation model (10) can be trained based on randomly rearranged data of the locations of 8 patches (labeled 1 to 8) out of 20 unmasked patches.

[0436] In step 3, the foundation model (10) can be trained based on randomly rearranged data of the locations of 13 patches (labeled 1 to 13) out of 20 unmasked patches.

[0437]

[0438] In this way, the first step may be steps up to the masking patch area determination step; masking step; encoding step; integrated feature information generation step; decoding step; and learning step; and each of the second and third steps may be steps up to the second masking patch area re-determination step; second re-masking step; second re-encoding step; second integrated feature information re-generation step; second re-decoding step; and second re-learning step.

[0439] In addition, the number of patches whose positions are rearranged in the second re-masking step may be greater than the number of patches whose positions are rearranged in the masking step.

[0440]

[0441] In addition, the steps up to the second masking patch area recreation step; the second re-masking step; the second re-encoding step; the second integrated feature information regeneration step; the second re-decoding step; and the second re-learning step may be repeated two or more times.

[0442] That is, the number of patches whose positions are rearranged in the second remasking step performed for the second time may be greater than the number of patches whose positions are rearranged in the first remasking step performed for the second time.

[0443]

[0444] In this way, the foundation model (10) can be trained by curriculum learning, which starts at a low difficulty level and gradually progresses to a higher difficulty level, so that it is trained efficiently and stably and reduces overfitting.

[0445]

[0446] FIG. 17 illustrates a method for training a foundation model (10) while gradually increasing noise according to one embodiment of the present invention.

[0447]

[0448] A method for learning a foundation model (10) for a hyperspectral image, comprising an artificial neural network-based encoder (100) and a decoder (101) as illustrated in FIG. 17, comprising: a masking patch area determination step for determining a masking patch area to be randomly masked for each of the multiple channels of a hyperspectral image having multiple channels according to wavelength, which is the subject of learning; a masking step for masking a masking patch area in each of the multiple channels of the hyperspectral image, inserting noise into at least some of the multiple patches that are not masked, and dividing the image into multiple patches; an encoding step for inputting the multiple patches divided into the artificial neural network-based encoder (100), excluding the masked portion of the hyperspectral image, to derive feature information for each of the multiple patches that are not masked; and an integrated feature information generation step for generating integrated feature information by inserting multiple masking feature information for the masked patches into the multiple feature information. A decoding step of generating a first reconstructed hyperspectral image by inputting the above integrated feature information into an artificial neural network-based decoder (101); a learning step of training an encoder (100) and a decoder (101) so as to minimize the difference between the hyperspectral image and the first reconstructed hyperspectral image; a third masking patch area recreation step of determining a masking patch area to be randomly masked for each of the multiple channels of a hyperspectral image having multiple channels according to wavelength that is the subject of learning; a third re-masking step of masking the masking patch area in each of the multiple channels of the hyperspectral image, inserting noise of a higher intensity than the noise inserted in the masking step into at least some of the multiple patches that are not masked, and dividing them into multiple patches; a third re-encoding step of inputting the divided multiple patches, excluding the masked part of the hyperspectral image, into an artificial neural network-based encoder (100) to derive feature information for each of the multiple patches that are not masked; A third integrated feature information regeneration step for generating integrated feature information by inserting a plurality of masking feature information for a patch masked in a plurality of the above feature information;The method comprises: a third re-decoding step of inputting the integrated feature information into an artificial neural network-based decoder (101) to generate a second reconstructed hyperspectral image; and a third re-learning step of training the encoder (100) and the decoder (101) so as to minimize the difference between the hyperspectral image and the second reconstructed hyperspectral image; wherein the integrated feature information may include, for each of the plurality of feature information and the plurality of masking feature information, information regarding a corresponding position in the hyperspectral image or information regarding the order between data.

[0449]

[0450] In one embodiment of the present invention, the foundation model (10) can be trained while gradually increasing the learning difficulty. Specifically, the foundation model (10) can be trained while inserting noise of higher intensity into the hyperspectral image to be trained.

[0451]

[0452] Specifically, in each of steps 1 to 3 illustrated in FIG. 17, the foundation model (10) can be learned through the process described in FIG. 13.

[0453] However, the intensity of noise inserted into multiple unmasked patches in the hyperspectral image being trained in each of steps 1 to 3 may differ.

[0454] Specifically, in step 1, the foundation model (10) is trained based on a hyperspectral image with low-intensity noise inserted into an unmasked patch, and in step 2, the foundation model (10) is trained based on a hyperspectral image with medium-intensity noise inserted into an unmasked patch, and can be trained based on a hyperspectral image with high-intensity noise inserted into an unmasked patch.

[0455]

[0456] In this way, the first step may be steps up to the masking patch area determination step; masking step; encoding step; integrated feature information generation step; decoding step; and learning step in the learning method of the foundation model (10), and the second and third steps may each be steps up to the third masking patch area re-determination step; third re-masking step; third re-encoding step; third integrated feature information re-generation step; third re-decoding step; and third re-learning step.

[0457] In addition, the intensity of the noise inserted in the third re-masking step may be stronger than the intensity of the noise inserted in the masking step.

[0458]

[0459] In addition, the steps up to the third masking patch area recombination step; the third re-masking step; the third re-encoding step; the third integrated feature information regeneration step; the third re-decoding step; and the third re-learning step may be repeated two or more times.

[0460] That is, the intensity of the noise inserted in the third re-masking step performed for the second time may be stronger than the intensity of the noise inserted in the third re-masking step performed for the first time.

[0461]

[0462] In this way, the foundation model (10) can be trained by curriculum learning, which starts at a low difficulty level and gradually progresses to a higher difficulty level, so that it is trained efficiently and stably and reduces overfitting.

[0463]

[0464] FIG. 18 illustrates patches that are rearranged by a plurality of channels of a hyperspectral image according to one embodiment of the present invention.

[0465]

[0466] As illustrated in FIG. 18, the masking step masks patches at the same location corresponding to the masking patch area in each channel of the hyperspectral image, and patches that are randomly mixed and rearranged among a plurality of unmasked patches can be rearranged to different locations in each channel of the hyperspectral image.

[0467]

[0468] As described above, the hyperspectral image may be data of a three-dimensional spectral structure containing pixel-by-pixel or patch-by-patch data for each of the multiple channels.

[0469] Meanwhile, in one embodiment of the present invention, patches corresponding to the same position in each of a plurality of channels in a hyperspectral image may be masked. For example, the 2nd, 5th, 6th, 7th, ..., 15th patches may be masked in each of Ch1, Ch2, ... and Chn.

[0470]

[0471] Meanwhile, in one embodiment of the present invention, among a plurality of unmasked patches in each of a plurality of channels in a hyperspectral image, patches that are randomly mixed and rearranged can be rearranged to different positions in each of the plurality of channels.

[0472] For example, among the unmasked 1st, 3rd, 4th, 8th, and 16th patches in Ch1, the patches can be randomly shuffled by rearranging the 1st patch to the 3rd, the 3rd patch to the 1st, the 4th patch to the 8th, the 8th patch to the 16th, and the 16th patch to the 4th.

[0473] On the other hand, among the unmasked 1st, 3rd, 4th, 8th, and 16th patches in Ch2, the patches can be randomly shuffled by rearranging the 1st patch to the 16th, the 3rd patch to the 8th, the 4th patch to the 3rd, the 8th patch to the 3rd, and the 16th patch to the 4th.

[0474]

[0475] In this way, in each of the multiple channels of the hyperspectral image, the masked patches may have the same position in each of the multiple channels, and the unmasked patches may have different positions in each of the multiple channels.

[0476]

[0477] Consequently, according to one embodiment of the present invention, the data size or amount of computation in the learning process can be reduced.

[0478] As described above, the integrated feature information includes a plurality of feature information and a plurality of masking feature information, each of which includes information about a corresponding position in the hyperspectral image or information about the order between the data.

[0479] Meanwhile, when the positions of the masked patch (or masking feature information for the corresponding patch) and the unmasked patch (or feature information generated for the corresponding patch) are all rearranged in each of the multiple channels of the hyperspectral image, the data size of the integrated feature information, or the amount of computation in the encoder (100) and decoder (101) may become excessively large, and the amount of computation or data storage required during the learning process may increase.

[0480]

[0481] In order to prevent such problems, the present invention can prevent the data size of the integrated feature information, or the amount of computation in the encoder (100) and decoder (101), from becoming excessively large by fixing the position of the masked patch and randomly rearranging only the position of the unmasked patch in each of the multiple channels of the hyperspectral image (without rearranging the positions of all patches regardless of whether they are masked).

[0482]

[0483] That is, the learning method of the present invention is capable of learning with high difficulty, with a large data size of integrated feature information and a large amount of computation in the encoder (100) and decoder (101) compared to a “learning method that does not rearrange the positions of all patches regardless of whether they are masked in each of the multiple channels of a hyperspectral image,” and conversely, the data size of integrated feature information and a small amount of computation in the encoder (100) and decoder (101) compared to a “learning method that rearranges the positions of all patches regardless of whether they are masked in each of the multiple channels of a hyperspectral image.”

[0484]

[0485] In another embodiment of the present invention, in order to train the foundation model (10) with the highest difficulty, the encoder (100) and decoder (101) can be trained based on a hyperspectral image in which the positions of all patches are rearranged regardless of whether they are masked in each of the multiple channels of the hyperspectral image.

[0486]

[0487] 3. Method for performing operations on hyperspectral images based on user requests

[0488]

[0489] A service server performing “3. Method for performing operations based on a user’s operation request regarding hyperspectral images” described below may include a foundation model generated or learned by “2. Method for generating a foundation model for hyperspectral images”.

[0490] Preferably, the encoder of the first task processing module included in the service server performing “3. Method for performing a task according to a user’s task request regarding a hyperspectral image” may be an encoder of a foundation model generated or learned by “2. Method for generating a foundation model for a hyperspectral image”.

[0491]

[0492] FIG. 19 illustrates a service server (1) that performs a method of performing a task according to a user's task request on a hyperspectral image according to one embodiment of the present invention.

[0493]

[0494] As illustrated in FIG. 19, the hyperspectral image includes information on the wavelength of light for each pixel according to a plurality of channels according to wavelength, and the encoder (100) may include an artificial neural network trained to extract feature information for the hyperspectral image.

[0495]

[0496] As illustrated in FIG. 19 (a), the service server (1) of the present invention communicates with a user terminal, receives a request for a task to be performed by the user terminal on a hyperspectral image, and can provide a task result obtained by performing the task.

[0497]

[0498] In one embodiment of the present invention, a user terminal can transmit a hyperspectral image to be worked on to a service server (1) along with a work request.

[0499] Alternatively, the service server (1) can store multiple hyperspectral images to be worked on and perform work on a hyperspectral image specified by a user terminal.

[0500]

[0501] In one embodiment of the present invention, the user terminal may be a computing device comprising one or more of a desktop, a laptop, a smartphone, and a tablet, comprising one or more processors and one or more memories.

[0502]

[0503] Meanwhile, the service server (1) may be connected to an externally built LLM or may include an LLM internally.

[0504] Specifically, LLM is a deep learning-based Large Language Model, and the service server (1) can derive work attribute information from a work request received from a user terminal using the LLM model. A detailed explanation of this will be provided later.

[0505] In one embodiment of the present invention, the LLM model may be a multimodal model. Specifically, the LLM model may receive not only text-based prompts but also various types of data such as documents, voice, video, images, etc.

[0506] For example, data including files such as documents, audio, video, and images uploaded by a user terminal can be input into the LLM model.

[0507] Alternatively, the hyperspectral image to be processed can be input into the LLM model.

[0508]

[0509] As illustrated in FIG. 19 (b), the service server (1) may include a plurality of components.

[0510] Specifically, the work request receiving unit (11) can receive a work request from a user terminal in which information to be performed on a hyperspectral image is entered in text form.

[0511] The work attribute derivation unit (12) can derive work attribute information for a work request by inputting the configured prompt into an LLM, the work request received from a user terminal, and a phrase requesting judgment of work attribute information required according to the work request.

[0512] Based on the information included in the above prompt, the LLM can determine whether the hyperspectral image for which the user terminal requested a task should be processed by the deep learning-based first task processing module (13) or the rule-based second task processing module (14), and in the present invention, the information can be defined as information on the main attributes of the task.

[0513] Additionally, based on the information included in the above prompt, LLM can determine which specific task corresponds to each case where processing is to be done in the first task processing module (13) or the second task processing module (14), and in the present invention, the information can be defined as information regarding the detailed attributes of the task.

[0514]

[0515] The service server (1) can determine which module, either the first task processing module (13) or the second task processing module (14), will process the work on the hyperspectral image based on the key attribute information derived by the LLM.

[0516] In addition, based on the detailed attribute information derived by the LLM, it can be determined which module among the plurality of first detailed work modules (130) of the first work processing module (13) will process the work on the hyperspectral image.

[0517] Alternatively, based on detailed attribute information derived by LLM, it can be determined which module among the plurality of second detailed task modules (140) of the second task processing module (14) will process the work on the hyperspectral image.

[0518]

[0519] The first task processing module (13) may include an encoder (100) and a plurality of first detailed task modules (130). Specifically, the encoder (100) is in a state where learning is completed according to the “2. Method for generating a foundation model for a hyperspectral image” and can extract feature information for the hyperspectral image to be processed.

[0520] Each of the plurality of first detailed work modules (130) may be a deep learning-based machine learning inference model trained to process the work performed by the corresponding first detailed work module (130) by receiving feature information extracted by inputting a hyperspectral image to be learned into the encoder (100).

[0521]

[0522] In one embodiment of the present invention, the first task processing module (13) or encoder (100) may be a BERT (Bidirectional Encoder Representations from Transformers) language model. Specifically, BERT may be a machine learning model based on an artificial neural network that is trained to divide an image into patches, vectorize (embed) them, and derive relationships between patches using embedding values ​​(feature information) for each patch.

[0523]

[0524] The second work processing module (14) may include a plurality of second detailed work modules (140). Specifically, each of the plurality of second detailed work modules (140) may receive a hyperspectral image to be worked on and perform the work performed by the corresponding second detailed work module (140).

[0525]

[0526] The result providing unit (15) can provide the work result for the hyperspectral image generated by the first work processing module (13) or the second work processing module (14) to the user terminal.

[0527]

[0528] FIG. 20 illustrates details regarding a work request and work attribute information according to an embodiment of the present invention.

[0529]

[0530] As illustrated in FIG. 20, a method for performing a task according to a user's task request regarding a hyperspectral image comprises: a task request reception step of receiving a task request in which information to be performed on the hyperspectral image is entered in text form from a user terminal; a task attribute information derivation step of configuring a prompt including the task request and a phrase requesting judgment of necessary task attribute information according to the task request, and inputting the prompt into an LLM to derive task attribute information regarding the task request; and a task execution step of determining, based on the task attribute information, which module the hyperspectral image should be processed by, either a deep learning-based first task processing module (13) or a rule-based second task processing module (14), and inputting the hyperspectral image into the determined first task processing module (13) or second task processing module (14) to derive a task result for the hyperspectral image. and a result providing step of providing the above work result to a user terminal; wherein the work attribute information may include information on a key attribute regarding which module, either a deep learning-based first work processing module (13) or a rule-based second work processing module (14), should process the hyperspectral image, and information on a detailed attribute of the work that is further distinguished from the key attribute.

[0531] In addition, the first detailed work module (130) may include one or more of the following: a learned machine learning inference model that super-resolves the hyperspectral image, a machine learning inference model including an artificial neural network that removes noise from the hyperspectral image, and a learned machine learning inference model that identifies and segments objects in the hyperspectral image.

[0532]

[0533] As illustrated in FIG. 20 (a), the service server (1) can receive a request for a task from a user terminal that includes information to be performed on a hyperspectral image.

[0534] Specifically, as illustrated in FIG. 20 (b), a work request may be data entered in text form by a user to perform a task on a hyperspectral image, such as ‘removing noise from the part presumed to be land, reducing noise in the area around the river, or super-resolution processing of vehicles located in the center.’

[0535]

[0536] Meanwhile, the types of tasks requested by users regarding hyperspectral images can be classified into tasks requiring deep learning-based operations, such as estimating parts corresponding to land, rivers, buildings, etc., or detecting abnormal spectral patterns, or tasks requiring rule-based operations, such as deleting data of specific channels (wavelengths) or deleting data of pixels with data values ​​higher than the average.

[0537]

[0538] The service server (1) can configure a prompt including the above work request and a phrase requesting the determination of necessary work attribute information according to the above work request, and input the above prompt into the LLM to derive work attribute information for the above work request.

[0539] Specifically, the service server (1) can input into the LLM a prompt that includes the above-mentioned work request and a phrase requesting the determination of the main attributes and detailed attributes of the work for the above-mentioned work request.

[0540] For example, the above phrase may include a phrase requesting to determine the main attributes of the task by determining whether the task according to the task request regarding the hyperspectral image should be processed by the deep learning-based first task processing module (13) or the rule-based (rule-based) second task processing module (14).

[0541] More preferably, the above phrase may include a phrase requesting to determine the detailed attributes of the task by determining which of the plurality of first detailed task modules (130) should process the task according to the task request regarding the hyperspectral image, or which of the plurality of second detailed task modules (140) should process the task according to the task request of the deep learning-based first task processing module (130).

[0542]

[0543] In one embodiment of the present invention, the prompt may include a phrase requesting a determination of whether additional information is required in addition to the hyperspectral image that is the target of the task when performing the task according to the task request, and if so, what additional information is required. When the service server (1) determines by the LLM that additional information is needed, it may request the additional information from the user terminal or retrieve the additional information from an external source.

[0544] For example, if a user wants to perform a task of distinguishing and displaying boundaries by administrative district in a hyperspectral image or inputting information about the use of a parcel (commercial area, residential area, agricultural area, etc.), the LLM may determine that additional parcel information including parcel boundaries, parcel number, area information, address information, owner information, and use information is required for the task, and the service server (1) may request parcel information from the user terminal or retrieve parcel information from external data.

[0545]

[0546] Such additional information can be additionally input to the first task processing module (13) or the second task processing module (14) that performs work on the hyperspectral image.

[0547]

[0548] As illustrated in (c) of FIG. 20, the work attribute information derived by the LLM for a work request may include information on the main attributes of the work, which distinguishes whether the work according to the work request is a 'work that must be processed by the first work processing module (13) of the artificial neural network method' or a 'work that must be processed by the first work processing module (13) of the rule-based method'.

[0549] In addition, the work attribute information derived by the LLM for a work request may include information on the detailed attributes of the work, such as which type of work the work is among the artificial neural network type work, when the work according to the work request is a 'work that must be processed by the first work processing module (13) of the artificial neural network type'.

[0550] In one embodiment of the present invention, information regarding detailed attributes of work attribute information may include one or more of super-resolution processing, noise removal, object identification, and segmentation.

[0551]

[0552] Each of the plurality of first detailed work modules (130) of the first work processing module (13) may be trained to perform work on the corresponding detailed attribute.

[0553] For example, the first sub-module (130) that performs super-resolution processing may be a machine learning inference model that includes an artificial neural network trained to perform super-resolution processing by receiving feature information extracted from inputting a hyperspectral image to be learned into an encoder (100). Alternatively, the first sub-module (130) that performs object identification and segmentation may be a machine learning inference model that includes an artificial neural network trained to identify and segment objects by receiving feature information extracted from inputting a hyperspectral image to be learned into an encoder (100).

[0554]

[0555] In addition, the work attribute information derived by the LLM for a work request may include information on the detailed attributes of the work, such as which type of work the work is among the rule-based work, when the work according to the work request is a 'work that must be processed by the second work processing module (14) of the rule-based method'.

[0556] In one embodiment of the present invention, information on detailed attributes of work attribute information may include one or more of the following operations: removing outliers that are three times higher than the average, removing pixels with a brightness of 50 or higher, or removing data with a wavelength of 400 nm or less.

[0557]

[0558] FIG. 21 illustrates details regarding a first work processing module (13) according to one embodiment of the present invention.

[0559]

[0560]

[0561] As illustrated in FIG. 21, the above-mentioned task execution step may include: a feature information extraction step in which, when the hyperspectral image needs to be processed in the first task processing module (13), the hyperspectral image is input into an artificial neural network-based encoder (100) included in the first task processing module (13) to extract feature information; and a task result derivation step in which the feature information is input into an artificial neural network-based first detailed task module (130) corresponding to the detailed attributes of the task attribute information included in the first task processing module (13) to derive a task result for the hyperspectral image.

[0562]

[0563] Figure 21 (a) illustrates that the task attribute information generated by the LLM is determined to be a super-resolution processing that must be processed by the first task processing module (13) of the artificial neural network method for processing the hyperspectral image.

[0564]

[0565] When the work attribute information is determined in this way, the service server (1) can input the hyperspectral image to be worked on into the encoder (100) of the first work processing module (13) to extract feature information.

[0566] Next, the service server (1) inputs the corresponding feature information into a learned first detailed task module (130) that performs super-resolution processing, and a work result in which super-resolution processing for a hyperspectral image is performed by the first detailed task module (130) can be produced.

[0567]

[0568] As such, according to one embodiment of the present invention, a configuration (encoder (100)) that compresses a hyperspectral image to generate feature information (regardless of the detailed attributes of the task) and a configuration (first detailed task module (130)) that performs actual work according to the detailed attributes of the task can be distinguished.

[0569] Consequently, compared to a method in which a process of compressing a hyperspectral image to extract feature information and performing actual work according to the detailed attributes of the work based on the extracted feature information is implemented in a single integrated module, each of the multiple first detailed work modules (130) can be lightweighted and optimized to be efficiently implemented.

[0570]

[0571] FIG. 22 illustrates details regarding a second work processing module (14) according to one embodiment of the present invention.

[0572]

[0573] As illustrated in FIG. 22, the second task processing module (14) includes one or more second detailed task modules (140) that perform tasks for each of one or more detailed attributes using a rule-based algorithm, and the task execution step may include a step of inputting the hyperspectral image into the second detailed task module (140) corresponding to the detailed attribute of the task attribute information to derive a task result when the hyperspectral image needs to be processed by the second task processing module (14).

[0574]

[0575] FIG. 22 (a) illustrates that the task attribute information generated by the LLM is determined to be 'a task to remove pixels with a brightness of 50 or higher' that must be processed by the rule-based second task processing module (14) for the hyperspectral image.

[0576]

[0577] When the work attribute information is determined in this way, the service server (1) inputs the hyperspectral image to be worked on into the second detailed work module (140) of the second work processing module (14) which performs the work of removing pixels with a brightness of 50 or higher, and the work result in which pixels with a brightness of 50 or higher are removed from the hyperspectral image can be produced by the second detailed work module (140).

[0578]

[0579] As such, according to one embodiment of the present invention, when the type of task requested by the user (main attributes and detailed attributes) is a relatively simple task that does not require deep learning processing and can be processed based on preset rules, the corresponding rules can be applied to the hyperspectral image to be worked on to derive a task result.

[0580]

[0581] FIG. 23 illustrates examples of operations performed on a hyperspectral image according to one embodiment of the present invention.

[0582]

[0583] As illustrated in FIG. 23 (a), the service server (1) can restore or extract components corresponding to soil, tree, and water, respectively, from the hyperspectral image to be worked on. Specifically, the corresponding task can be processed by the first task processing module (13) of the artificial neural network method.

[0584]

[0585] As illustrated in FIG. 23 (b), the service server (1) can identify and segment objects identified as aircraft in the hyperspectral image to be worked on. Specifically, the work can be processed by the first work processing module (13) of the artificial neural network method.

[0586]

[0587] Preferably, the work illustrated in each of (a) and (b) of FIG. 23 can be processed by different first detailed work modules (130) of the first work processing module (13).

[0588]

[0589] FIG. 24 schematically illustrates the internal configuration of a computing device according to one embodiment of the present invention.

[0590]

[0591] The service server illustrated in FIG. 1 described above may include the components of the computing device (11000) illustrated in FIG. 24.

[0592] As illustrated in FIG. 24, the computing device (11000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600). In this case, the computing device (11000) may correspond to the service server illustrated in FIG. 1.

[0593] The memory (11200) may include, for example, high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or various other data required for the operation of the computing device (11000).

[0594] At this time, access to memory (11200) from other components, such as the processor (11100) or peripheral device interface (11300), can be controlled by the processor (11100).

[0595] The peripheral device interface (11300) can connect input and / or output peripheral devices of the computing device (11000) to the processor (11100) and memory (11200). The processor (11100) can perform various functions for the computing device (11000) and process data by executing software modules or instruction sets stored in the memory (11200).

[0596] The input / output subsystem can connect various input / output peripherals to the peripheral interface (11300). For example, the input / output subsystem may include a controller for connecting peripherals such as a monitor, keyboard, mouse, printer, or, if necessary, a touchscreen or sensor to the peripheral interface (11300). According to another aspect, input / output peripherals may be connected to the peripheral interface (11300) without passing through the input / output subsystem.

[0597] The power circuit (11500) can supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include one or more power sources such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0598] The communication circuit (11600) can enable communication with another computing device using at least one external port.

[0599] Alternatively, as described above, the communication circuit (11600) may enable communication with other computing devices by including an RF circuit and transmitting and receiving an RF signal, also known as an electromagnetic signal.

[0600] The embodiment of FIG. 24 is merely an example of a computing device (11000), and the computing device (11000) may have some components shown in FIG. 24 omitted, additional components not shown in FIG. 24 added, or a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may include a touchscreen or sensors, etc., in addition to the components shown in FIG. 24, and the communication circuit (11600) may include a circuit for RF communication of various communication methods (WiFi, 3G, LTE, Bluetooth, NFC, Zigbee, etc.). The components that can be included in the computing device (11000) may be implemented as hardware, software, or a combination of both hardware and software, including one or more integrated circuits specialized for signal processing or applications.

[0601] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a computing device (11000) through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file in response to a request from the computing device (11000).

[0602]

[0603] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0604] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computing devices and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0605] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0606]

[0607] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below are also within the scope of the claims.

Claims

1. A method for performing operations on hyperspectral images in accordance with a user's operation request, wherein A task request reception step of receiving a task request from a user terminal in which information to be performed on a hyperspectral image is entered in text form; A step for deriving work attribute information, comprising: configuring a prompt including the above work request and a phrase requesting the determination of necessary work attribute information according to the above work request; and inputting the above prompt into an LLM to derive work attribute information for the above work request; A task execution step of determining, based on the above task attribute information, whether the hyperspectral image should be processed by a deep learning-based first task processing module or a rule-based second task processing module, and inputting the hyperspectral image into the determined first task processing module or second task processing module to derive a task result for the hyperspectral image; and A result provision step for providing the above work result to a user terminal; is included, The above work attribute information is, It includes information on key attributes regarding which module, either a deep learning-based first task processing module or a rule-based second task processing module, should process the above hyperspectral image, and information on detailed attributes of the task that are further distinguished from the said key attributes. The above work execution step is, When the hyperspectral image needs to be processed in the first task processing module, a feature information extraction step of inputting the hyperspectral image into an artificial neural network-based encoder included in the first task processing module to extract feature information; and A method for performing a task according to a user's task request, comprising: a task result derivation step of inputting the above feature information into a first detailed task module based on an artificial neural network corresponding to a detailed attribute of the task attribute information included in the first task processing module to derive a task result for the hyperspectral image.

2. In Claim 1, The above second task processing module is, A rule-based algorithm comprising one or more second sub-task modules that perform operations on each of one or more detailed attributes, and The above work execution step is, A method for performing a task according to a user's task request, comprising the step of inputting the hyperspectral image into the second detailed task module corresponding to the detailed attribute of the task attribute information when the hyperspectral image needs to be processed in the second task processing module, and deriving a task result.

3. In Claim 1, The above-mentioned first detailed work module is, A method for performing a task according to a user's task request, comprising one or more of a learned machine learning inference model that super-resolves the hyperspectral image, a machine learning inference model including an artificial neural network that removes noise from the hyperspectral image, and a learned machine learning inference model that identifies and segments objects in the hyperspectral image.

4. In Claim 1, The above hyperspectral image is, It includes information on the wavelength of light for each pixel, for each of the multiple channels according to wavelength, and The above encoder is, A method for performing a task according to a user's task request, comprising an artificial neural network trained to extract feature information for hyperspectral images.

5. In Claim 1, The above encoder is trained by performing the following steps, and The steps below above are, A mask patch area determination step for determining a mask patch area to be randomly masked for each of the multiple channels for a hyperspectral image having multiple channels according to wavelength, which is a learning target; A masking step of masking a masking patch region in each of the plurality of channels of the hyperspectral image and dividing it into a plurality of patches; An encoding step of inputting a plurality of segmented patches, excluding the masked portion of the hyperspectral image, into an artificial neural network-based encoder to derive first feature information for each of the plurality of unmasked patches; An integrated feature information generation step of generating integrated feature information by inserting a plurality of masking feature information for a patch masked in a plurality of the above-mentioned first feature information; A decoding step of inputting the above integrated feature information into an artificial neural network-based decoder to generate a reconstructed hyperspectral image; and A learning step for training an encoder and a decoder to minimize the difference between the hyperspectral image and the reconstructed hyperspectral image; is included. The above integrated feature information is, A method for performing a task according to a user's task request, wherein each of the plurality of first feature information and the plurality of masking feature information includes information about a corresponding position in the hyperspectral image or information about the order between data.

6. In Claim 5, The above masking step is, In each of the multiple channels of the above hyperspectral image, Fix the positions of multiple masked patches, and A method for performing an operation according to a user's operation request, comprising the step of rearranging at least some of the unmasked patches by randomly shuffling their positions.

Images