Method and device for drug classification by multi-vision artificial intelligence
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-08-13
Smart Images
Figure KR2025023287_13082026_PF_FP_ABST
Abstract
Description
Drug classification method and device of multi-vision artificial intelligence
[0001] The present invention was carried out under the support of the Ministry of Health and Welfare under project unique number 240004386 and project number 02223382, the research management agency for the above project is the Korea Health Industry Development Institute, the research project name is "Development of a pilot model for medical artificial intelligence technology based on multi-institutional-multimodal federated learning", the research task name is "Demonstration of FLAME-ARK solution development: Construction of a hospital-specific federated learning-based multimodal foundation model and development of an AI solution", the lead institution is Samsung Seoul Hospital, and the research period is 2025-04-01~2025-12-31.
[0002] The present invention relates to an automated drug classification method and system that classifies drugs by combining hyperspectral images and RGB images, and more specifically, to data preprocessing and artificial intelligence-based classification technology that enables accurate and efficient classification by simultaneously utilizing the visual and spectral characteristics of drugs.
[0003]
[0004] Drug classification is an essential procedure in the manufacturing, distribution, storage, and prescription of medications, and accuracy in this process is directly linked to patient safety. Existing drug classification methods primarily rely on visual identification or traditional computer vision techniques using single RGB images. However, these methods are limited in classification accuracy due to the visual similarity of drugs, and they are particularly limited in distinguishing between drugs that are similar in color, shape, and texture.
[0005] Hyperspectral imaging technology is gaining attention to address these issues. Hyperspectral imaging collects data across a broad spectral band, including not only visible light but also infrared and ultraviolet regions, to provide information on the chemical properties and structural characteristics of substances. This enables the identification of subtle differences in drugs that are difficult to distinguish using RGB images alone.
[0006] Existing hyperspectral image-based classification systems have primarily been implemented by utilizing single spectral information or by independently analyzing hyperspectral and RGB images and then comparing the results. However, this approach has limitations in that it fails to fully leverage the additional benefits that can be obtained from the interaction and combination between hyperspectral and RGB images.
[0007] This invention proposes a new data processing and classification method that combines hyperspectral images and RGB images to simultaneously utilize spatial and spectral information. Through this, the invention aims to overcome the limitations of existing technologies and significantly improve the accuracy and efficiency of drug classification.
[0008]
[0009] Therefore, the present invention aims to overcome the limitations of existing methods and to classify drugs accurately and efficiently by combining RGB images and hyperspectral images to simultaneously utilize visual and spectral information of drugs.
[0010]
[0011] To achieve the above objective, the present invention comprises an automated drug classification method for classifying drugs by combining hyperspectral images and RGB images, wherein the method comprises: a collection step of collecting hyperspectral images and RGB images of drugs through a hyperspectral camera and an RGB camera; a camera calibration step of matching the point of view of the hyperspectral camera and the RGB camera in pixel units using checkerboard calibration; a preprocessing step of labeling drug class information corresponding to the calibrated hyperspectral images and RGB images, adjusting the number of image channels for combining the hyperspectral images and RGB images, and emphasizing spatial information; a combining step of combining the preprocessed hyperspectral images and RGB images using a weighted channel combination method; a feature extraction step of extracting spectral and spatial features from the combined images through a feature extraction model; and a classification step of classifying drugs by inputting the extracted features into a classification model.
[0012] Preferably, the checkerboard correction may be characterized by obtaining a transformation matrix based on the intrinsic characteristics and relative positions of the hyperspectral camera and the RGB camera for matching points between the two cameras by utilizing at least one checkerboard shape element among the grid shape, corner points, and consistent spacing of the checkerboard.
[0013] Preferably, the preprocessing step may be characterized by applying principal component analysis (PCA) to the hyperspectral image to reduce the number of channels and retain core spectral information when preprocessing the corrected hyperspectral image.
[0014] Preferably, the preprocessing step may be characterized by, in preprocessing the RGB image, generating a single-channel image with emphasized spatial features using at least two of histogram equalization, Canny edge detection, Sobel transform, local binary pattern (LBP), and grayscale transformation on the RGB image, and combining the single-channel images generated by the plurality of techniques to form a composite-channel image combined with multiple spatial features.
[0015] Preferably, the weighted channel combination method may be characterized by forming input data in which multiple channels are combined by dynamically adjusting the importance of spectral features and spatial features by assigning channel-specific weights using 1x1 convolution to each channel of the preprocessed hyperspectral and RGB images.
[0016] Preferably, the weights assigned through the 1x1 convolution are learnable parameters that are updated using backpropagation and optimization algorithms during the neural network learning process, thereby dynamically reflecting the importance between spectral and spatial feature channels.
[0017] Preferably, the feature extraction model may be characterized by including at least one of a 2D convolutional neural network (2D CNN), a 3D convolutional neural network (3D CNN), a hybrid CNN combining 2D and 3D convolution, or a SpectralNet for spectral specialized analysis.
[0018] Preferably, the classification model may be characterized by using a K-Nearest Neighbors (KNN) model to measure distance-based similarity between the features of the labeled images corresponding to a specific class and the extracted features, based on the labeled images to which label information was assigned in the preprocessing step, and classifying drugs by utilizing a plurality of the labeled images closest to the extracted features.
[0019] In addition, the present invention is further characterized by comprising: an automated drug classification system that classifies drugs by combining hyperspectral images and RGB images, wherein the system includes: a collection unit that collects hyperspectral images and RGB images of drugs through a hyperspectral camera and an RGB camera; a camera calibration unit that aligns the point of view of the hyperspectral camera and the RGB camera in pixel units using checkerboard calibration; a preprocessing unit that labels drug class information corresponding to the calibrated hyperspectral images and RGB images, adjusts the number of image channels for combining different types of images, and emphasizes spatial information; a combining unit that integrates the preprocessed hyperspectral images and RGB images using a weighted channel combination method and combines them into an input data form; a feature extraction unit that extracts spectral and spatial features from the combined input data through a feature extraction model; and a classification unit that classifies drugs by inputting the extracted features into a classification model.
[0020]
[0021] According to the present invention, by combining RGB images and hyperspectral images to simultaneously analyze the visual and spectral information of a drug, there is an advantage of providing higher accuracy than existing classification methods.
[0022]
[0023] Figure 1 shows a flowchart of an automated drug classification method according to an embodiment of the present invention.
[0024] Figure 2 shows the wavelength of an image that can be collected in the collection step according to an embodiment of the present invention.
[0025] FIG. 3 illustrates a method for matching the point of view of a hyperspectral camera and an RGB camera at the pixel level according to an embodiment of the present invention.
[0026] Figure 4 shows a flowchart of a preprocessing process for optimizing spectrum bands in hyperspectral data processing according to an embodiment of the present invention.
[0027] FIG. 5 shows a flowchart of a preprocessing process for generating more useful input data by utilizing spatial information of an RGB image according to an embodiment of the present invention.
[0028] Figure 6 shows a flowchart of the preprocessing process of an RGB image in the preprocessing step according to an embodiment of the present invention.
[0029] FIG. 7 illustrates two methods for combining the channels of a preprocessed hyperspectral image and an RGB image according to an embodiment of the present invention.
[0030] FIG. 8 shows a framework of an automated drug classification method according to an embodiment of the present invention.
[0031] FIG. 9 schematically shows the block configuration of a computer device (100) according to one embodiment of the present disclosure.
[0032] FIG. 10 schematically illustrates the computing environment of a computer device (100) according to one embodiment of the present disclosure.
[0033]
[0034] An automated drug classification method for classifying drugs by combining hyperspectral images and RGB images, comprising: a collection step of collecting hyperspectral images and RGB images of drugs through a hyperspectral camera and an RGB camera; a camera calibration step of matching the point of view of the hyperspectral camera and the RGB camera in pixel units using checkerboard calibration; a preprocessing step of labeling drug class information corresponding to the calibrated hyperspectral images and RGB images, adjusting the number of image channels for combining the hyperspectral images and RGB images, and emphasizing spatial information; a combining step of combining the preprocessed hyperspectral images and RGB images using a weighted channel combination method; a feature extraction step of extracting spectral and spatial features from the combined images through a feature extraction model; and a classification step of classifying drugs by inputting the extracted features into a classification model.
[0035]
[0036] The present invention will be described in detail below with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by exemplary embodiments. Identical reference numerals in each drawing indicate components that perform substantially the same function.
[0037] The purpose and effects of the present invention may be naturally understood or become clearer through the following description, and the purpose and effects of the present invention are not limited solely to the description below. Furthermore, in describing the present invention, if it is determined that a detailed description of known technology related to the present invention may unnecessarily obscure the essence of the present invention, such detailed description will be omitted.
[0038] The terms used in this invention are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the description of the invention, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0039] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.
[0040] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this invention.
[0041] In interpreting the components, they are interpreted to include a margin of error even without a separate explicit indication. In the case of descriptions regarding temporal relationships, for example, where the temporal sequence is described using 'after,' 'following,' 'next,' 'before,' etc., cases that are not continuous are included unless 'immediately' or 'directly' is used.
[0042] Hereinafter, the technical configuration of the present invention will be described in detail with reference to the attached drawings.
[0043] FIG. 1 shows a flowchart of an automated drug classification method according to an embodiment of the present invention. Referring to FIG. 1, the automated drug classification method may include a collection step (S100), a camera calibration step (S110), a preprocessing step (S120), a combination step (S130), a feature extraction step (S140), and a classification step (S150).
[0044] The automated drug classification method combines hyperspectral and RGB images to automate drug classification. Images collected by hyperspectral and RGB cameras are preprocessed by aligning viewpoints through checkerboard calibration and applying PCA and spatial feature enhancement, respectively. The automated drug classification method integrates spectral and spatial data using a weighted channel combination method, extracts spectral and spatial features using a CNN, and classifies drugs using a KNN model.
[0045] The automated drug classification method may include a collection step (S100) for collecting hyperspectral images and RGB images of a drug through a hyperspectral camera and an RGB camera. The automated drug classification method may include a camera calibration step (S110) for matching the point of view of the hyperspectral camera and the RGB camera at the pixel level using checkerboard calibration. The automated drug classification method may include a preprocessing step (S120) for labeling drug class information corresponding to the calibrated hyperspectral images and RGB images, adjusting the number of image channels for combining the hyperspectral images and RGB images, and emphasizing spatial information. The automated drug classification method may include a combining step (S130) for combining the preprocessed hyperspectral images and RGB images using a weighted channel combining method. The automated drug classification method may include a feature extraction step (S140) for extracting spectral and spatial features from the combined images through a feature extraction model. The automated drug classification method may include a classification step (S150) for classifying drugs by inputting extracted features into a classification model.
[0046] The collection step (S100) can collect hyperspectral images and RGB images of the drug through a hyperspectral camera and an RGB camera. Specifically, the collection step (S100) can collect hyperspectral images and RGB images of the drug through a hyperspectral camera and an RGB camera using two different cameras.
[0047] The RGB image collected in the collection step (S100) includes a major band in the visible light region, and each wavelength may have a blue wavelength of 450–510 nm, a green wavelength of 510–580 nm, and a red wavelength of 630–690 nm.
[0048] The hyperspectral image collected in the collection step (S100) may contain data composed of continuous spectral bands corresponding to wavelengths. The hyperspectral image has wavelengths in the range of 400 nm to 3000 nm and provides various information through the relationships between spectral bands, but some bands may overlap, which may cause problems that reduce data processing efficiency. Overlapping spectral bands may result in unnecessary calculations and slow down the processing speed. In the collection step (S100), the hyperspectral image may be replaced with a multispectral image and a superspectral image.
[0049] In one embodiment, FIG. 2 shows the wavelengths of an image that can be collected in the collection step (S100) according to an embodiment of the present invention. Referring to FIG. 2, the collection step (S100) can collect multispectral images and superspectral images, and the wavelengths of the multispectral images may have a Coastal Blue wavelength of 400~450 nm, a blue wavelength of 450~510 nm, a green wavelength of 510~580 nm, a yellow wavelength of 585 nm~625 nm, a red wavelength of 630~690 nm, a Red Edge wavelength of 705~745 nm, and a Near-Infrared 1 wavelength of 770~895 nm. The wavelengths of the superspectral image are Coastal Blue 400–450 nm, Blue 450–510 nm, Green 510–580 nm, Yellow 585–625 nm, Red 630–690 nm, Red Edge 705–745 nm, Near-Infrared 1 770–895 nm, Near-Infrared 2 860–1040 nm, Short-Wave Infrared 1 1195–1225 nm, Short-Wave Infrared 2 1550–1590 nm, and Short-Wave Infrared 3. It can have wavelengths in the range of 1640~1680nm, Short-Wave Infrared 4 wavelength 1710~1750nm, Short-Wave Infrared 5 wavelength 2145~2185nm, Short-Wave Infrared 6 wavelength 2185~2225nm, and Short-Wave Infrared 7 wavelength 2235~2285nm.Hyperspectral images include a superspectral region and may additionally have a short-wave infrared wavelength region of 2285–2400 nm, and a mid-wave infrared region and a long-wave infrared region.
[0050] FIG. 3 illustrates a method for aligning the point of view of a hyperspectral camera and an RGB camera at the pixel level according to an embodiment of the present invention. Referring to FIG. 3, the camera calibration step (S110) can align the point of view of the hyperspectral camera and the RGB camera at the pixel level using checkerboard calibration. In the camera calibration step (S110), the checkerboard calibration utilizes at least one checkerboard shape element among the grid shape, corner points, and consistent spacing of the checkerboard to obtain a transformation matrix for the matching points between the two cameras based on the unique characteristics and relative positions of the hyperspectral camera and the RGB camera. In the camera calibration step (S110), the transformation matrix in the checkerboard calibration can accurately align the point of view of the camera, ensure that each pixel of the hyperspectral image and the RGB image corresponds accurately to a pixel of the other image, and enable the analysis of the hyperspectral image and the RGB image for the same scene.
[0051] In one embodiment, the camera calibration step (S110) may include a light-based approach, a hardware-based approach, a software-based approach, and a hardware-software mix depending on the approach method. In the camera calibration step (S110), the light-based approach is a method of simultaneously transmitting optical signals of the same scene to an RGB camera and a hyperspectral camera using a beam splitter. In the camera calibration step (S110), the light-based approach enables the simultaneous capture of RGB and hyperspectral images and can be implemented with a simple structure. In the camera calibration step (S110), the hardware-based approach is a method of firmly fixing the RGB camera and the hyperspectral camera using rigid mounting. In the camera calibration step (S110), the hardware-based approach is designed to accurately align the two cameras and enable stable operation through robust equipment. In the camera calibration step (S110), the hardware-based approach is characterized by being insensitive to changes in the external environment and being effective for long-term shooting in a fixed state. In the camera calibration step (S110), the software-based approach captures RGB images and hyperspectral images separately, and then aligns and fuses the two images through a post-processing process. In the camera calibration step (S110), the software-based approach synchronizes the captured data using a software algorithm and generates a final fused image through the data matching and alignment process. In the camera calibration step (S110), the software-based approach reduces hardware dependency and can be flexibly applied in various environments.In the camera calibration step (S110), the hardware-software hybrid approach is a method that performs hardware alignment and software calibration simultaneously using checkerboard calibration. The alignment state between the RGB camera and the hyperspectral camera is measured based on a checkerboard pattern, and the measured state is corrected software-wise to improve data accuracy. This method can generate high-precision fused data by combining the physical stability of alignment with the flexibility of software calibration. As an example, the approach of the camera calibration step (S110) is not limited to the method mentioned above, and applicable known calibration methods may be used.
[0052] FIG. 4 shows a flowchart of a preprocessing process for optimizing spectrum bands in hyperspectral data processing according to an embodiment of the present invention. Referring to FIG. 4, the preprocessing step (S120) preprocesses the corrected hyperspectral image by applying principal component analysis (PCA) to the hyperspectral image to reduce the number of channels and retain core spectral information.
[0053] Specifically, the hyperspectral image preprocessing corrected in the preprocessing step (S120) is performed using Principal Component Analysis (PCA), which can reduce the vast number of spectral channels of the hyperspectral image. Reducing the number of channels in the preprocessing step (S120) may involve dimensionality reduction. Although a hyperspectral image can have infinitely many channels, many of these channels have high correlations and contain redundant information.
[0054] Preprocessing the hyperspectral image with PCA in the preprocessing step (S120) converts the hyperspectral image into a new set of uncorrelated features (principal components), enabling the capture of the largest variance with a small number of components.
[0055] Preprocessing hyperspectral images with PCA in the preprocessing step (S120) can reduce the computational burden of feature extraction models including CNNs by reducing the number of channels, thereby speeding up training and inference times. In one embodiment, reducing the number of channels to 3 allows for maintaining compatibility with RGB-based CNN architectures.
[0056] Preprocessing the hyperspectral image with PCA in the preprocessing step (S120) can improve the signal-to-noise ratio by discarding less useful information while retaining components representing the most important information, and in particular, can reduce noisy or irrelevant information in some bands of the hyperspectral image, and enable effective feature extraction.
[0057] FIG. 5 illustrates a flowchart of a preprocessing process for generating more useful input data by utilizing spatial information of an RGB image according to an embodiment of the present invention. Referring to FIG. 5, the preprocessing step (S120) preprocesses the RGB image by using at least two techniques among histogram equalization, Canny edge detection, Sobel transform, local binary pattern (LBP), and grayscale transformation on the RGB image to generate a single-channel image with emphasized spatial features, and can combine the single-channel images generated by the plurality of techniques to form a composite-channel image combined with multiple spatial features. The techniques for preprocessing the RGB image in the preprocessing step (S120) include histogram equalization, Canny edge detection, Sobel transform, local binary pattern, and grayscale transformation, but are not limited thereto and various preprocessing techniques may be applied.
[0058] Specifically, in the preprocessing step (S120), RGB image preprocessing can enhance features of the RGB image obtained in the data collection step (S100) by undergoing various transformations, similar to the preprocessing applied to hyperspectral images. The purpose of these various techniques is to emphasize spatial information that may not be prominent in hyperspectral data. Each transformation technique can emphasize different spatial features. Preprocessing the RGB image with histogram equalization in the preprocessing step (S120) has the characteristic of redistributing pixel intensity to adjust image contrast, thereby making features more distinguishable, and increasing the pixel intensity distribution to make fine details in low-contrast areas stand out. Preprocessing the RGB image with Canny edge detection in the preprocessing step (S120) has the characteristic of identifying object boundaries by detecting areas with rapid intensity changes in the image, and defining new boundaries using a multi-stage algorithm including Gaussian filtering, gradient calculation, and non-maximum suppression. Preprocessing the RGB image with the Sobel transform in the preprocessing step (S120) is a technique that detects edges and contours by calculating changes in pixel intensity in horizontal and vertical directions, and is characterized by providing edges emphasized along the slope and clarifying spatial information. Preprocessing the RGB image with a local binary pattern in the preprocessing step (S120) is a technique that encodes the local texture of the image, assigns binary values based on the brightness and darkness of surrounding pixels, emphasizes fine texture patterns, and distinguishes delicate spatial details.
[0059] The preprocessing step (S120) may include other techniques for preprocessing RGB images, emphasizing specific types of spatial features and capturing details that may be lacking in hyperspectral data, thereby enabling more comprehensive analysis. Each transformation in the preprocessing step (S120) for preprocessing RGB images generates a single-channel image, and combining these channels can generate a composite image encoding multiple spatial features, which is utilized as an enhanced input for subsequent processing or analysis.
[0060] In one embodiment, FIG. 6 shows a flowchart of the preprocessing process of an RGB image in a preprocessing step (S120) according to an embodiment of the present invention. Referring to FIG. 6, it shows that a 3-channel image is generated by combining each single-channel image generated in the preprocessing step (S120) using a direct channel mapping method. The 3-channel image generated from the RGB channels in the preprocessing step (S120) may be subjected to a direct channel mapping method that reflects the information of each single channel as is, a weighted composite method that combines by applying weights to each channel, an operation combination method that combines by applying mathematical operations (sum, product, difference) to each channel, or a stack method that repeats one channel, but is not limited thereto.
[0061] FIG. 7 illustrates two methods for combining channels of a preprocessed hyperspectral image and an RGB image according to an embodiment of the present invention. Referring to FIG. 7, the combining step (S130) includes direct combination and weighted combination. In the combining step (S130), X channels (X_Channels) can be defined as channels of the preprocessed hyperspectral image and Y channels (Y_Channels) as channels of the preprocessed RGB image.
[0062] In the combination step (S130), direct combination combines the X channel and the Y channel with equal weight, all channels have equal importance, no specific channel is emphasized more, and is simple and computationally efficient, but it may not be suitable when specific information is more important.
[0063] In the combination step (S130), weighted combination is performed by assigning different weights (W1, W2) to the X channel and the Y channel, respectively. The weights of a specific channel or group of channels can be changed according to their importance, and a combination optimized for a specific application (e.g., when spectral information is more important) is possible. Additionally, the weighted combination can be used as weights during the training process of an artificial intelligence model.
[0064] Specifically, the combining step (S130) can form input data in which multiple channels are combined by dynamically adjusting the importance of spectral features and spatial features by assigning channel-specific weights using 1x1 convolution to each channel of the preprocessed hyperspectral and RGB images. Assigning channel-specific weights using 1x1 convolution to each channel in the combining step (S130) may utilize weighted combining.
[0065] In the combination step (S130), the weights assigned through 1x1 convolution are learnable parameters that are updated using backpropagation and optimization algorithms during the neural network learning process, thereby dynamically reflecting the importance between spectral and spatial feature channels.
[0066] FIG. 8 illustrates a framework of an automated drug classification method according to an embodiment of the present invention. Referring to FIG. 8, the feature extraction step (S140) extracts spectral and spatial features from an integrated image through a feature extraction model, and the integrated image may be a weighted combined image. In the feature extraction step (S140), the feature extraction model may include at least one of a 2D convolutional neural network (2D CNN), a 3D convolutional neural network (3D CNN), a hybrid CNN combining 2D and 3D convolution, or a SpectralNet for spectral-specialized analysis, but the feature extraction model is not limited thereto.
[0067] In one embodiment, the feature extraction step (S140) may apply one or more of a 2D convolutional neural network (2D CNN), a 3D convolutional neural network (3D CNN), a hybrid CNN that combines 2D and 3D convolution, or a SpectralNet for spectral-specialized analysis in order to efficiently extract spectral and spatial features by receiving a weighted combined image as input.
[0068] In one embodiment of the feature extraction step (S140), a 2D convolutional neural network (2D CNN) model may be input with an image in which hyperspectral bands are reduced to a certain number (e.g., 3 to n) through a preprocessing process and spatial channels extracted from an RGB image are combined. This 2D convolutional method is simple to implement and requires relatively little computation because it can utilize structures widely used in the field of computer vision (e.g., ResNet, VGG, etc.). However, since the 2D filter only indirectly handles the interaction between each band, it may be desirable to apply additional preprocessing or extension techniques when distinguishing drugs with similar spectral characteristics.
[0069] In one embodiment of the feature extraction step (S140), a 3D convolutional neural network (3D CNN) model may apply a 3D convolutional filter by considering the hyperspectral image in the form of (horizontal x vertical x spectral channels). Through this, spectroscopic-spatial information can be reflected simultaneously by learning not only the spatial dimension but also the channel (band) axis at once. In particular, since the reflection and absorption rates may differ in different wavelength bands even for drugs that appear visually similar, the 3D CNN can effectively identify these detailed spectral differences. However, compared to a 2D CNN, the number of parameters and computational load increase, and the hardware burden must be considered in environments requiring large datasets or real-time processing.
[0070] In one embodiment of the feature extraction step (S140), a hybrid CNN model combines the advantages of a 3D CNN and a 2D CNN. It applies 3D convolution in the initial layer to broadly capture interactions between spectral channels and spatial information, and then uses 2D convolution in the later layer to precisely extract spatial details such as edges, textures, and shapes. This allows for reducing the computational burden compared to using only a 3D CNN while maintaining spectro-spatial fusion characteristics. In fact, hybrid CNNs are known in numerous studies as models that simultaneously satisfy high accuracy and relatively low computational load, and in the embodiment of FIG. 8, they can serve as an alternative advantageous for improving drug classification performance.
[0071] In one embodiment of the feature extraction step (S140), the SpectralNet reduces the number of channels or extracts multiscale spectral information through PCA, Factor Analysis (FA), wavelet transform, etc., to solve high-dimensional problems of hyperspectral data, and then inputs this into a 2D CNN structure (SpectralNet). This reduces computational load because it does not require directly processing the entire band with 3D convolution, and allows for learning spatial features while maintaining core spectral characteristics extracted from multiple bands. Although RGB channels can be combined in the preprocessing stage of the SpectralNet, there are cases where the input format must be modified to fit the structure. Nevertheless, by appropriately combining dimensionality reduction and multi-resolution analysis, there is an advantage in that classification accuracy can be increased to the level of a 3D CNN.
[0072] Referring to FIG. 8, the classification step (S150) can classify drugs by inputting extracted features into a classification model. The classification model uses a K-Nearest Neighbors (KNN) model that measures distance-based similarity between the features of labeled images corresponding to a specific class and the extracted features, based on labeled images to which label information was assigned in the preprocessing step (S120), and can classify drugs by utilizing multiple labeled images closest to the extracted features. However, this is merely one embodiment, and the classification model used in the classification step (S150) is not limited thereto.
[0073] Specifically, the classification step (S150) can classify spectral and spatial features obtained during the CNN-based feature extraction process through a KNN classifier. The classification model used in the classification step (S150) is a non-parametric, instance-based classifier that can classify new samples through distance-based comparison with pre-trained data. The classification model does not require assumptions about the distribution of the data and can perform classification by measuring the similarity between the extracted features and the labeled data as distance and referencing multiple closest example data. Since the classification model can reflect subtle visual differences and effectively distinguish drugs based on unique spectral and spatial features, it can be an ideal classifier for tasks requiring precision, such as drug classification. Furthermore, unlike fully connected layers, the classification model does not require iterative training or backpropagation, making it computationally efficient and capable of reducing memory and processing requirements. These advantages of the classification model make it suitable for real-world applications requiring fast and accurate drug classification. The classification model provides an efficient and reliable approach for classifying drugs by utilizing previously extracted spectral-spatial features, overcoming the limitations of existing classification methods. The non-parametric nature of the classification model and its case-based approach enable rapid and flexible classification when new data is presented.
[0074]
[0075] Example 1: Performance Comparison Based on Feature Extraction Model
[0076]
[0077] The following describes, as a specific embodiment of the present invention, the results of applying a 2D convolutional neural network (2D CNN), a 3D convolutional neural network (3D CNN), a hybrid CNN, and a SpectralNet, respectively, in the feature extraction step (S140) after preprocessing by combining hyperspectral images and RGB images. In this embodiment, for representative hyperspectral public datasets such as Indian Pines (224 spectral bands, 16 classes), University of Pavia (103 spectral bands, 9 classes), and Salinas Scene (224 spectral bands, 16 classes), the experimental results of setting the training sample ratio to 10% or 30% are summarized in [Table 1].
[0078] Training SamplesMethodsIP datasetUP datasetSA datasetOAKappaAAOAKappaAAOAKappaAA10%SVM81.67±0.6578.76±0.7779.84±3.3790.58±0.4787.21±0.7092.99±0.3694.46±0.1293.13±0.3493.01±0.602D-CNN80.27±1.278.26±2.168.32±4.196.63±0.295.53±0.294.84±1.496.34±0.395.93±0.994.36±0.53D-CNN82.62±0.179.25±0.376.51±0.196.34±0.294.90±1.297.03±0.685.00±0.183.20±0.789.63±0.2M3D-CNN81.39±2.681.20±2.075.22±0.795.95±0.693.40±0.497.52±1.094.20±0.893.61±0.396.66±0.5FuSENet97.11±0.297.25±0.297.32±0.297.65±0.397.69±0.397.68±0.499.23±0.199.97±0.299.16±0.1HybridSN98.39±0.198.16±0.198.01±0.299.72±0.199.64±0.199.20±0.199.98±0.299.98±0.299.98±0.1SpectralNET98.76±0.298.59±0.198.61±0.199.71±0.199.62±0.199.43±0.299.96±0.199.96±0.199.97±0.130%SVM87.24±0.3885.27±0.4585.15±1.1095.65±0.1394.63±0.1794.60±0.1494.95±0.1094.48±0.1197.93±0.112D-CNN88.90±1.387.01±1.685.70±1.096.50±0.496.55±0.396.00±0.196.75±0.696.71±0.798.57±0.23D-CNN90.23±0.289.70±0.389.87±0.197.90±0.397.22±0.197.30±0.195.54±0.594.81±0.397.09±0.6M3D-CNN95.67±0.194.70±0.394.60±0.697.60±0.296.50±0.698.00±0.194.99±0.395.40±0.196.28±0.2FuSENet99.01±0.298.60±0.198.64±0.199.42±0.299.21±0.399.33±0.299.68±0.299.74±0.199.69±0.1HybridSN99.75±0.199.71±0.199.63±0.299.98±0.199.98±0.299.97±0.2100100100SpectralNET99.86±0.299.84±0.299.98±0.199.99±0.199.98±0.199.98±0.1100100100.
[0079] First, looking at an example of applying a 2D convolutional neural network (2D CNN), hyperspectral bands are reduced to three or several through PCA, and spatial information extracted from RGB channels is combined and input into a 2D convolution (based on ResNet-18, etc.). Referring to [Table 1], if only 10% of the training samples from the Indian Pines dataset are used, the OA is measured as 80.27±1.2%, Kappa as 78.26±2.1, and AA as 68.32±4.1, and if increased to 30%, the OA improves to 88.90±1.3%, Kappa to 87.01±1.6, and AA to 85.70±1.0. In the University of Pavia dataset, OA 96.63±0.2%, Kappa 95.53±0.2, and AA 94.84±1.4 were achieved with 10% training samples, and when increased to 30%, OA 96.50±0.4%, Kappa 96.55±0.3, and AA 96.00±0.1 were improved. Salinas Scene also shows OA 96.34±0.3%, Kappa 95.93±0.9, and AA 94.36±0.5 when trained at 10%, and improves to OA 96.75±0.6%, Kappa 96.71±0.7, and AA 98.57±0.2 when scaled up to 30%. Secondly, 3D Convolutional Neural Networks (3D CNNs) treat hyperspectral images as 3D tensors in the form of (horizontal x vertical x bands) to learn spatial and spectral information simultaneously. While this approach demonstrates higher accuracy compared to 2D CNNs, it has the disadvantage of increasing the number of parameters and computational load. As shown in [Table 1], on the Indian Pines dataset, OA is measured at 82.62±0.1%, Kappa at 79.25±0.3, and AA at 76.51±0.1 when training at 10%, and when increased to 30%, these values improve to OA 90.23±0.2%, Kappa 89.70±0.3, and AA 89.87±0.1. In the University of Pavia and Salinas scenes as well, applying 3D CNN shows an accuracy improvement of more than 1–5% compared to 2D CNN.Third, the Hybrid SN is a structure that combines the advantages of 3D CNN and 2D CNN. It applies 3D convolution in the initial layer to learn spectral and spatial information extensively, and then reinforces spatial details through 2D convolution in subsequent layers. As shown in [Table 1], in Indian Pines, OA was measured at 98.39±0.2%, Kappa at 98.16±0.1%, and AA at 98.01±0.2% with only 10% training. When the training was increased to 30%, high accuracy was achieved, reaching OA 99.75±0.1%, Kappa 99.71±0.1%, and AA 99.63±0.2. The Hybrid SN also demonstrates high performance in the University of Pavia and Salinas scenes, and among the models in the examples, it exhibits the highest accuracy at 10% training.
[0080] Finally, SpectralNET is an approach that takes note of the large number of hyperspectral bands, reduces and analyzes the bands into wavelets using PCA or wavelet transform, and then trains them using a 2D CNN-based network. Referring to [Table 1], when training with 10% on Indian Pines, OA was recorded as 98.76±0.2%, Kappa as 98.59±0.1, and AA as 98.61±0.1; when training with 30%, OA was achieved as 99.86±0.2%, Kappa as 99.84±0.2, and AA as 99.98±0.1, demonstrating classification performance that is virtually at the highest level. In other datasets such as Salinas Scene, it also has the highest accuracy among the models in the examples by combining dimensionality reduction and multiscale processing.
[0081] Although the present invention has been described in detail above through representative embodiments, those skilled in the art will understand that various modifications can be made to the above-described embodiments within the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the claims set forth below as well as all modifications or variations derived from the claims and equivalent concepts.
[0082] FIG. 9 schematically illustrates the block configuration of a computer device (100) according to one embodiment of the present disclosure. The computer device (100) according to the present embodiment may include a processor (110) and a memory (130).
[0083] The configuration of the computer device (100) illustrated in FIG. 9 is merely a simplified example. In this embodiment, the computer device (100) may additionally include other components for performing a computing environment, and the computer device (100) may be configured with only some of the disclosed configurations.
[0084] Additionally, the computer device (100) in this specification may be used interchangeably with a computer device and is interpreted to encompass any type of server or any type of terminal. The computer device (100) of this disclosure may refer to any type of component constituting a system for implementing the embodiments.
[0085] The computer device (100) may refer to a user terminal of any form or a server of any form, and the components of the computer device (100) described above are merely examples, so some may be excluded or additional components may be included. For example, if the computer device (100) includes a user terminal, an output unit (not shown) and an input unit (not shown) may be included within that scope.
[0086] The computer device (100) in this specification can perform various technical features according to the embodiments of the present disclosure described below.
[0087] In one embodiment, the processor (110) may be composed of at least one core and may include various types of processors for data analysis and / or processing, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computer device (100).
[0088] The processor (110) can read a computer program stored in memory (130) and provide a benchmark result according to one embodiment of the present disclosure.
[0089] According to one embodiment of the present disclosure, a processor (110) can perform operations for learning a neural network. The processor (110) can perform various operations necessary for learning a neural network, such as processing input data for learning during a deep learning process, extracting features from input data, calculating errors, and updating the weights of the neural network through backpropagation. At least one of the CPU, GPGPU, and TPU included in the processor (110) can process the learning of a network function, and, for example, the CPU and GPGPU may together perform the learning of a network function or data classification using said network function. In addition, in this embodiment, it is also possible to process the learning of a network function or data classification using a network function by utilizing processors included in a plurality of computer devices together. Furthermore, a computer program executed in a computer device (100) according to one embodiment of the present disclosure may be a program executable on a CPU, GPGPU, or TPU.
[0090] Additionally, the processor (110) can generally handle the overall operation of the computer device (100). For example, the processor (110) can process data, information, signals, etc. that are input or output through various components included in the computer device (100), or execute an application program stored in a storage unit to provide appropriate information or functions to the user.
[0091] According to one embodiment of the present disclosure, the collection unit (120) can collect hyperspectral images and RGB images of a drug through a hyperspectral camera and an RGB camera. The collection unit (100) can perform the aforementioned collection step (S100).
[0092] According to one embodiment of the present disclosure, the memory (130) may store various forms of information generated or determined by the processor (110) and various forms of information received by the computer device (100). Additionally, the memory (130) may be a storage medium for storing computer software that enables the processor (110) to perform operations according to the embodiment of the present disclosure. Accordingly, the memory (130) may encompass software code required to perform the embodiment of the present disclosure, data to be executed by said code, and a computer reading medium for storing the result of code execution.
[0093] According to one embodiment of the present disclosure, memory (130) may refer to any type of storage medium. For example, memory (130) may include at least one storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Additionally, the computer device (100) may operate in conjunction with web storage that performs the storage function of memory (130) over the internet. The description of memory described above is merely an example, and the memory (130) used in the present disclosure is not limited to such examples.
[0094] The communication unit (not shown) in the present disclosure may be configured regardless of the type of communication method, such as wired or wireless, and may include various types of communication networks, such as a Personal Area Network (PAN) and a Wide Area Network (WAN). Additionally, the network unit (150) may operate based on the known World Wide Web (WWW) and may also utilize wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth.
[0095] The computer device (100) in the present disclosure may include any type of user terminal and / or any type of server. Accordingly, embodiments of the present disclosure may be performed by a server and / or a user terminal.
[0096] In one embodiment, the user terminal may include any type of terminal capable of interacting with a server or other computer device. Examples of user terminals include mobile phones, smartphones, laptop computers, personal digital assistants (PDAs), slate PCs, tablet PCs, ultrabooks, etc.
[0097] In one embodiment, the server may include any type of computing system or computer device, such as, for example, a microprocessor, a mainframe computer, a digital processor, a portable device, a device controller, etc.
[0098] The storage unit may be contained within the server or exist under the server's management. As another example, the storage unit may be implemented to exist outside the server while being able to communicate with it. In this case, the storage unit may be managed and controlled by an external server distinct from the server. Furthermore, if the storage unit exists outside the server and is implemented to communicate with it, an external server different from the server may likewise be responsible for the management and control of the storage unit.
[0099] FIG. 10 schematically illustrates the computing environment of a computer device (100) according to one embodiment of the present disclosure.
[0100] Components, modules, or units in this specification may include routines, procedures, programs, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will fully recognize that the methods presented in this disclosure may be implemented in various computer system configurations, such as single-processor or multi-processor computer devices, minicomputers, mainframe computers, as well as personal computers, handheld computer devices, microprocessor-based or programmable consumer electronics. Each of these may operate in connection with one or more associated devices.
[0101] The embodiments described in this disclosure may also be implemented in a distributed computing environment in which various tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
[0102] Computer devices typically include various computer-readable media. Any medium accessible by a computer can be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. For example, computer-readable media may include both computer-readable storage media and computer-readable transmission media.
[0103] The computer device (100) can be used in the same way as a computing device.
[0104] Computer-readable storage media include volatile and non-volatile media, transient and non-transient media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data.
[0105] Computer-readable storage media may include RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, and may also include any other media that can be accessed by a computer and used to store desired information, provided, but not limited thereto.
[0106] Computer-readable transmission media generally implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism, and include all information transmission media. Here, a modulated data signal refers to a signal in which one or more of its characteristics are set or modified to encode information within the signal.
[0107] For example, computer-readable transmission media may include wired media such as wired networks or direct-wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also included within the scope of computer-readable transmission media.
[0108] An exemplary environment (1000) implementing various aspects of the present invention including a computer (1100) is illustrated, wherein the computer (1100) includes a processing unit (1110), a system memory (1200), and a system bus (1101). The computer (1100) in this specification may be used interchangeably with the computer device (100).
[0109] The system bus (1101) connects system components, including system memory (1200) (but not limited thereto), to the processing unit (1110). The processing unit (1110) may be any of the various commercial processors, and dual processors and other multiprocessor architectures may also be used as the processing unit (1110).
[0110] The system bus (1101) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. The system memory (1200) includes read-only memory (ROM) (1210) and random access memory (RAM) (1220).
[0111] The Basic Input / Output System (BIOS) is stored in non-volatile memory (1210), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines used to transfer information between components within the computer (1100) in situations such as during startup. Additionally, RAM (1220) may include high-speed RAM, such as static RAM, for caching data.
[0112] The computer (1100) may also include an internal hard disk drive (HDD) (1300) (e.g., EIDE, SATA), a magnetic floppy disk drive (FDD) (1310) (e.g., for reading or writing from a removable diskette (1311)), an SSD, and an optical disk drive (1320) (e.g., for reading a CD-ROM disk (1322) or reading or writing from other high-capacity optical media such as a DVD).
[0113] A hard disk drive (1300), a magnetic disk drive (1310), and an optical disk drive (1320) can each be connected to a system bus (1101) through a hard disk drive interface (1330), a magnetic disk drive interface (1340), and an optical drive interface (1350).
[0114] Additionally, the interface (1330) for implementing an external drive may include, for example, at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.
[0115] These drives and associated computer-readable media provide non-volatile storage, such as data, data structures, computer-executable instructions, etc. In the case of a computer (1100), these drives and media are used to store any data in a suitable digital format. Although removable optical media such as HDDs, removable magnetic disks, CDs, or DVDs are mentioned as computer-readable storage media described above, those skilled in the art will understand that various types of computer-readable storage media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments. It will also be well recognized that any of these storage media may contain computer-executable instructions for carrying out the methods of the present invention.
[0116] Various program modules, such as an operating system (1400), one or more application programs (1410), other program modules (1420), and program data (1430), may be stored in the drive and RAM (1220). All or part of the operating system, application, module and / or data may also be cached in RAM (1220). Those skilled in the art will be fully aware that the present invention may be implemented in various commercially available operating systems or in combination of multiple operating systems.
[0117] The user can input commands or information into the computer (1100) through one or more wired or wireless input devices, such as a pointing device like a keyboard (1500) and a mouse (1510). Other input devices (not shown) may include various devices such as a microphone, an IR remote control, a joystick, a game pad, a stylus pen, and a touch screen. These input devices are usually connected to the processing unit (1110) through an input device interface (1520) connected to the system bus (1101), but they may also be connected through other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, and an IR interface.
[0118] A monitor (1530) or other types of display devices are also connected to the system bus (1101) via an interface such as a video adapter (1540). In addition to the monitor (1530), the computer may generally include other peripheral output devices (not shown), such as speakers, printers, etc.
[0119] A computer (1100) may operate in a networked environment by being logically connected to one or more remote computers (2048) via wired and / or wireless communication. The remote computers (2048) may be workstations, server computers, routers, personal computers, portable computers, microprocessor-based entertainment devices, peer devices, or other general network nodes, and may include many or all of the components generally described for the computer (1100), but for simplicity, only the memory storage device (1610) is shown. The logical connection may include wired / wireless connections to a local area network (LAN) (1700) and / or a larger network, e.g., a wide area network (WAN) (1710). These LAN and WAN networking environments are commonly used in office or corporate environments and can easily implement enterprise-wide computer networks such as intranets, all of which can be connected to a global computer network, e.g., the Internet.
[0120] When used in a LAN networking environment, the computer (1100) is connected to a local network (1700) via a wired and / or wireless communication network interface or adapter (1720). The adapter (1720) may facilitate wired or wireless communication to the LAN (1700), and the LAN (1700) may also include a wireless access point installed to communicate with the wireless adapter (1720). When used in a WAN networking environment, the computer (1100) may include a modem (1730), be connected to a communication server on the WAN (1710), or have other means of communicating through the WAN (1710), such as via the Internet. The modem (1730) may be an internal or external, wired or wireless device and is connected to the system bus (1101) via a serial port interface (1520). In a networked environment, the program modules described for the computer (1100) or parts thereof may be stored in a remote memory / storage device (1610). It is also well known that the illustrated network connection is exemplary and that other means of establishing a communication link between computers may be used.
[0121] The computer (1602) may operate with any wireless device or object connected via wireless communication, such as a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, various equipment or places associated with a wireless detectable tag, and a device that communicates with a telephone. Such wireless communication includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication method may be a predefined structure as in a conventional network, or it may be ad hoc communication between at least two devices.
[0122] It should be understood that the specific order or hierarchy of the steps within the presented process is an example of an exemplary approach. Depending on design priorities, the specific order or hierarchy of the steps within the process may be rearranged within the scope of this disclosure. The method claims of this disclosure present elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy derived therefrom.
[0123]
[0124]
[0125]
[0126] Therefore, the present invention aims to overcome the limitations of existing methods and to classify drugs accurately and efficiently by combining RGB images and hyperspectral images to simultaneously utilize visual and spectral information of drugs.
Claims
1. A drug classification method in which a computing device combines hyperspectral images and RGB images to classify drugs, A collection step for collecting hyperspectral images and RGB images of a drug through a hyperspectral camera and an RGB camera; A camera calibration step for matching the point of view of the hyperspectral camera and the RGB camera at the pixel level using checkerboard calibration; A preprocessing step for labeling drug class information corresponding to the corrected hyperspectral image and RGB image, adjusting the number of image channels for combining the hyperspectral image and RGB image, and emphasizing spatial information; A combining step that integrates preprocessed hyperspectral images and RGB images using a weighted channel combining method; A feature extraction step for extracting spectral and spatial features from an integrated image through a feature extraction model; and A classification step that classifies drugs by inputting extracted features into a classification model; An automated drug classification method characterized by including 2. In Paragraph 1, The above checkerboard calibration is, An automated drug classification method characterized by obtaining a transformation matrix based on the unique characteristics and relative positions of a hyperspectral camera and an RGB camera, utilizing at least one checkerboard shape element among the grid shape, corner points, and consistent spacing of the checkerboard.
3. In Paragraph 1, The above preprocessing step is, In preprocessing the above-mentioned corrected hyperspectral image, An automated drug classification method characterized by applying principal component analysis (PCA) to the hyperspectral image to reduce the number of channels and retain core spectral information.
4. In Paragraph 1, The above preprocessing step is, In preprocessing the above RGB image, An automated drug classification method characterized by generating a single-channel image with spatial features emphasized using at least two of histogram equalization, Canny edge detection, Sobel transform, local binary pattern (LBP), and grayscale transformation on the above RGB image, and combining the single-channel images generated by the plurality of techniques to form a composite-channel image combined with multiple spatial features.
5. In Paragraph 1, The above weighted channel combination method is, An automated drug classification method characterized by forming input data combining multiple channels by dynamically adjusting the importance of spectral features and spatial features by assigning channel-specific weights using 1x1 convolution to each channel of the preprocessed hyperspectral and RGB images.
6. In Paragraph 5, The weights assigned through the above 1x1 convolution are learnable parameters, During the neural network training process, it is updated using backpropagation and optimization algorithms, and An automated drug classification method characterized by dynamically reflecting the importance between spectral and spatial feature channels through this.
7. In Paragraph 1, The above feature extraction model is, An automated drug classification method characterized by including at least one of a 2D convolutional neural network (2D CNN), a 3D convolutional neural network (3D CNN), a hybrid CNN combining 2D and 3D convolution, or a SpectralNet for spectral specialized analysis.
8. In Paragraph 1, The above classification model is, Based on the labeled image to which label information was assigned in the above preprocessing step, A K-Nearest Neighbors (KNN) model is used to measure distance-based similarity between the features of the labeled images corresponding to a specific class and the extracted features, An automated drug classification method characterized by classifying drugs using a plurality of labeled images closest to the extracted features.
9. In a computing device for classifying drugs, The above computing device is, A collection unit including a hyperspectral camera and an RGB camera; At least one processor; and Memory that stores instructions executable by the above processor Includes, By executing the above command, the above processor, Hyperspectral images and RGB images of the drug are collected through the hyperspectral camera and the RGB camera, and Camera calibration is performed using checkerboard calibration to align the viewpoints between the hyperspectral camera and the RGB camera at the pixel level, and Based on drug class information corresponding to the corrected hyperspectral image and the RGB image, preprocessing is performed to adjust the number of image channels and emphasize spatial information for combining the hyperspectral image and the RGB image, and An integrated image is generated by integrating the preprocessed hyperspectral image and the RGB image using a weighted channel combination method, and Spectral features and spatial features are extracted from the above integrated image through a feature extraction model, and Configured to classify drugs by inputting extracted features into a classification model, Computing device.