Method and system for integrating convolutional neural network and lightweight artificial intelligence for on-device image classification

Abstract

Provided are a method and system for integrating a CNN with lightweight artificial intelligence for on-device image classification. An on-device integrated artificial intelligence system, according to an embodiment of the present invention, comprises: a first network which is a non-lightweight network that extracts features from an input image; and a second network which is a lightweight network that performs inference on the input image on the basis of the extracted features. Accordingly, the performance of the lightweight artificial intelligence can be complemented by integrating the feature point extraction function of the CNN into a lightweight network in an on-device environment.

Description

Method and System for Integrating Convolutional Neural Networks and Lightweight Artificial Intelligence for On-Device Image Classification

[0001] The present invention relates to an on-device artificial intelligence system, and more specifically, to a method for improving the performance of lightweight artificial intelligence in an on-device environment.

[0002] With the rapid advancement of artificial intelligence technology in recent years, research aimed at applying it to various fields is actively underway. As AI technology based on massive language models demonstrates task performance capabilities superior to humans in some areas, it is expected that AI will be used in even more fields in the near future than it is today. While such massive model-based AI excels at performing complex and abstract tasks, it requires very high power consumption and high-performance computing resources. Conversely, since running massive models is impossible on edge-level devices constrained by processing power and power consumption, lightweight AI technologies—which are structurally simpler and offer advantages in terms of power consumption and memory usage—are primarily applied.

[0003] Lightweight AI technologies include algorithm and network lightweighting to efficiently utilize existing models, but there are also methods for developing lightweight AI algorithms characterized by lower hardware resource and power consumption. In particular, lightweight algorithms that perform image classification using feature points extracted from data as input can operate within a single network structure, requiring low hardware resources and making them suitable for environments where maximum performance is achieved at a low cost.

[0004] However, since the performance of these lightweight algorithms depends on the characteristics of the feature points extracted from the data, the application of appropriate feature point extraction techniques is crucial. Feature point extraction techniques require high technical expertise as they must consider both temporal and frequency domains depending on the characteristics of the data. In particular, for image data, there is a problem in applying feature point extraction techniques because even data containing the same object exhibits very different characteristics depending on the degree of tilt or the position of the object.

[0005] The present invention has been devised to solve the aforementioned problems, and the objective of the present invention is to provide an integrated artificial intelligence system that performs CNN-based feature extraction on an input image and performs lightweight artificial intelligence-based image inference on the extracted feature points, by integrating the feature extraction function of a CNN (Convolutional Neural Network) into a lightweight network deployed in an on-device environment as a method to complement the performance of lightweight artificial intelligence.

[0006] An on-device integrated artificial intelligence system according to one embodiment of the present invention for achieving the above objective comprises: a first network that extracts features from an input image as a non-lightweight network; and a second network that performs inference about the input image from the extracted features as a lightweight network.

[0007] The first network may be a Convolutional Neural Network (CNN) comprising an input layer, at least one convolution layer, and at least one Fully Connected (FC) layer.

[0008] The second network may include at least one of a lightweight neural network, a quantized network, a pruned network, and a knowledge distillation network.

[0009] Inference may include at least one of image classification, object detection, object tracking, and image segmentation.

[0010] The first network and the second network can be installed on the edge device.

[0011] The second network may be capable of on-device learning.

[0012] The first network may not be learned on-device.

[0013] On-device learning can be on-device learning for images of a new category.

[0014] According to another aspect of the present invention, an on-device integrated artificial intelligence operation method is provided, characterized in that a first network, which is a non-lightweight network, extracts features from an input image; and a second network, which is a lightweight network, performs inference about the input image from the extracted features.

[0015] As described above, according to the embodiments of the present invention, by integrating the feature extraction function of a CNN into a lightweight network in an on-device environment, performing CNN-based feature extraction on an input image, and performing lightweight AI-based image inference on the extracted feature points, it is possible to improve the performance of lightweight AI.

[0016] FIG. 1 is an on-device integrated artificial intelligence system according to one embodiment of the present invention,

[0017] Figure 2 shows the structure of an RCE-NN that can be used as a lightweight artificial intelligence network.

[0018] FIG. 3 is an on-device integrated artificial intelligence operation method according to another embodiment of the present invention,

[0019] Figures 4 and 5 show simulation results using the MNIST dataset for performance evaluation of an on-device integrated artificial intelligence system,

[0020] FIG. 6 is a diagram illustrating the configuration of an on-device integrated artificial intelligence system according to another embodiment of the present invention.

[0021] The present invention will be described in more detail below with reference to the drawings.

[0022] An embodiment of the present invention presents a method and system for integrating CNN and lightweight artificial intelligence for on-device image classification. This technology integrates the feature extraction function of a CNN into a lightweight network deployed in an on-device environment to perform CNN-based feature extraction on an input image and perform lightweight artificial intelligence-based image classification on the extracted feature points. Furthermore, on-device training is additionally performed on the lightweight artificial intelligence to enable image classification for new categories.

[0023] FIG. 1 is a diagram illustrating the configuration of an on-device integrated artificial intelligence system according to an embodiment of the present invention. The on-device integrated artificial intelligence system according to an embodiment of the present invention may be mounted on an edge device platform, but may also be mounted on other levels of devices. The on-device integrated artificial intelligence system according to an embodiment of the present invention is configured to include a CNN (110) and a lightweight artificial intelligence network (130) as illustrated.

[0024] CNN (110) is a deep learning network that extracts feature points from an input image, and refers to a non-lightweight network or a processor that drives this network. The feature point data of the input image extracted by the CNN (110) becomes input data for a lightweight artificial intelligence network (130).

[0025] As illustrated, CNN (110) is composed of an input layer (Input), a number of convolution layers (Conv 1 to Conv N), and a number of fully connected layers (FC 1 to FC M).

[0026] Since the excellent feature extraction performance of CNN (110) has been proven in various fields such as image classification and object detection, the embodiment of the present invention is implemented to extract feature points of an image using CNN (110) in an on-device environment, even though it is a non-lightweight network.

[0027] Meanwhile, unlike the lightweight artificial intelligence network (130) described later, no additional on-device training is performed for the CNN (110). This is because training the CNN (110) on edge-level devices can result in a large overload.

[0028] A lightweight artificial intelligence network (130) is a network that classifies input images from feature point data extracted from a CNN (110), and unlike a CNN (110), it refers to a non-lightweight network or a processor that drives this network.

[0029] A lightweight artificial intelligence network (130) can be implemented as a Restricted Coulomb Energy-Neural Network (RCE-NN). Figure 2 shows the structure of an RCE-NN that can be used as a lightweight artificial intelligence network (130). An RCE-NN is a supervised learning-based network that performs image classification by training to analyze patterns of input feature point data and cluster them. It determines the class of input data by comparing the distance between the input data and the neurons of the hidden layer and evaluating whether the input is within the Active Influence Field (AIF) of each input. Then, based on the features of the input data and the distance to the neurons constituting the RCE-NN, the image classification is assigned to the class of the closest neuron. Therefore, it is important to extract features from the input data that accurately represent the characteristics of each class.

[0030] A lightweight artificial intelligence network (130) is suitable for use on edge-level devices due to its low computational complexity and on-device learning capabilities. However, the lightweight artificial intelligence network (130) may have poor image classification performance. Extracting suitable features from data of each class is difficult and time-consuming, and especially in the case of image data, features can vary depending on factors such as the orientation, color, and position of an object even within the same class, making it difficult to apply an appropriate feature extraction algorithm. Accordingly, in an embodiment of the present invention, feature point data for image classification is extracted using a CNN (110), which has already proven its feature point extraction performance in various fields.

[0031] Meanwhile, the lightweight artificial intelligence network (130) is trained using feature point data extracted by the CNN (110), and can perform image classification for new categories through additional on-device training.

[0032] The operation of the presented on-device integrated artificial intelligence system will be described in detail below with reference to FIG. 3. FIG. 3 is a diagram illustrating the flow of an on-device integrated artificial intelligence operation method according to another embodiment of the present invention.

[0033] As described above, the operation mode of the integrated artificial intelligence system is first determined (S210). If it is in inference mode (S210-inference mode), the CNN (110) extracts feature points from the input image (S220), and the lightweight artificial intelligence network (130) classifies the input image from the feature point data extracted in step S220 (S230) and outputs the classification result (S240).

[0034] On the other hand, in the case of an on-device learning mode (S210-learning mode), the CNN (110) receives an input image of a new category and extracts feature points (S250), and the lightweight artificial intelligence network (130) classifies the input image from the feature point data extracted in step S220 and backpropagates the error of the classification result to perform learning of the lightweight artificial intelligence network (130) (S260), thereby updating the lightweight artificial intelligence network (130) (S240). By doing so, image classification by the lightweight artificial intelligence network (130) becomes possible even for images of a new category.

[0035] A simulation using the MNIST dataset for evaluating the performance of an on-device integrated artificial intelligence system according to an embodiment of the present invention will be described in detail below with reference to FIGS. 4 and FIGS. 5.

[0036] An image classification system using only CNN (hereinafter referred to as "CNN") was implemented with the configuration shown in Fig. 4, and an on-device integrated artificial intelligence system according to an embodiment of the present invention (hereinafter referred to as "CNN + RCE") was implemented by connecting RCE-NN to the first FCL of the CNN shown in Fig. 4.

[0037] And as shown in Figure 5, when training and testing were performed without excluding specific numbers from the MNIST dataset, the accuracy of CNN and CNN + RCE was 99.08% and 96.48%, respectively, with CNN showing a slightly higher accuracy. Also, when training and testing were performed by excluding specific numbers '0' or '1' from the MNIST dataset, the accuracy of CNN was higher than that of CNN + RCE.

[0038] However, when trained on the NIST dataset excluding specific numbers '0' or '1' and then tested without excluding those numbers, the accuracy of CNN + RCE was found to be higher than that of CNN alone, confirming that CNN + RCE demonstrates performance more suitable for on-device environments.

[0039] FIG. 6 is a diagram illustrating the configuration of an on-device integrated artificial intelligence system according to another embodiment of the present invention. The on-device integrated artificial intelligence system according to an embodiment of the present invention differs from the system presented in FIG. 1 above in that it further includes an intermediate connection network (120) in addition to the CNN (110) and the lightweight artificial intelligence network (130).

[0040] The intermediate connection network (120) refers to a network that connects the output of the CNN (110) and the input of the lightweight artificial intelligence network (130), or a processor that drives this network. The intermediate connection network (120) may be implemented as a Multi-Layer Perceptron (MLP) or as an encoding network that connects multiple convolution layers.

[0041] Up to now, preferred embodiments of a method and system integrating CNN and lightweight artificial intelligence for on-device image classification have been described in detail.

[0042] In the above embodiment, as a method to complement the performance of lightweight artificial intelligence, an integrated artificial intelligence system is presented that integrates the feature extraction function of a CNN into a lightweight network deployed in an on-device environment, performs CNN-based feature extraction on an input image, and performs lightweight artificial intelligence-based image classification on the extracted feature points.

[0043] Meanwhile, the RCE-NN presented in the above embodiment is mentioned as an example of a lightweight artificial intelligence network (130). Therefore, the lightweight artificial intelligence network (130) can be implemented as a lightweight network such as a quantized network, a pruned network, or a knowledge distillation network, as well as other types of lightweight networks in addition to the RCE-NN.

[0044] Meanwhile, the image classification presented in the above embodiment is also mentioned as one of the inference functions of the lightweight artificial intelligence network (130). Therefore, it goes without saying that the technical concept of the present invention can be applied even when the lightweight artificial intelligence network (130) performs object detection, object tracking, image segmentation, etc.

[0045] Meanwhile, it goes without saying that the technical concept of the present invention may also be applied to a computer-readable recording medium containing a computer program that enables the device and method according to the present embodiment to perform their functions. Furthermore, the technical concept according to various embodiments of the present invention may be implemented in the form of computer-readable code recorded on a computer-readable recording medium. A computer-readable recording medium may be any data storage device that can be read by a computer and store data. For example, a computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, etc. Additionally, computer-readable code or a program stored on a computer-readable recording medium may be transmitted through a network connected between computers.

[0046] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

Claims

1. A first network that extracts features from an input image as a non-lightweight network; and An on-device integrated artificial intelligence system characterized by including a second network that performs inference on an input image from extracted features as a lightweight network.

2. In Claim 1, The first network is, An on-device integrated artificial intelligence system characterized by being a Convolutional Neural Network (CNN) comprising an input layer, at least one convolution layer, and at least one Fully Connected (FC) layer.

3. In Claim 1, The second network is, An on-device integrated artificial intelligence system characterized by comprising at least one of a lightweight neural network, a quantized network, a pruned network, and a knowledge distillation network.

4. In Claim 1, Reasoning is, An on-device integrated artificial intelligence system characterized by including at least one of image classification, object detection, object tracking, and image segmentation.

5. In Claim 1, The first network and the second network are, An on-device integrated artificial intelligence system characterized by being installed on an edge device.

6. In Claim 5, The second network is, An on-device integrated artificial intelligence system characterized by the ability to perform on-device learning.

7. In Claim 6, The first network is, An on-device integrated artificial intelligence system characterized by not being learned on-device.

8. In Claim 6, On-device learning is, An on-device integrated artificial intelligence system characterized by on-device learning of images of a new category.

9. A first network, which is a non-lightweight network, extracts features from an input image; and An on-device integrated artificial intelligence operation method characterized by including the step of a second network, which is a lightweight network, performing inference on an input image from extracted features.

Images