Model optimization system based on cross-distillation for data scarcity and method thereof

Abstract

The invention relates to a system and method thereof for improving the performance of computer vision models in cases of data scarcity by transferring knowledge from compressed domain (HEVC) data to the pixel domain by cross-distillation.

Description

[0001] MODEL OPTIMIZATION SYSTEM BASED ON CROSS-DISTILLATION FOR DATA SCARCITY AND METHOD THEREOF

[0002] Technical Field of the Invention

[0003] The invention relates to a system and method thereof for improving the performance of computer vision models in cases of data scarcity by transferring knowledge from compressed domain (HEVC) data to the pixel domain by cross-distillation.

[0004] State of the Art

[0005] Data scarcity is a major problem in computer vision applications, especially when sufficient training data is not available due to confidentiality, cost, or limited resources. In current applications, knowledge transfer is generally done within the same data domain and these methods generally consist of models trained with pixel-based images. However, knowledge transfer between different data modalities, such as compressed domain knowledge, is underutilized. This limits the learning capacity of the model and leads to a loss of performance in data scarcity scenarios.

[0006] In current techniques, the method of cross-distillation makes it possible to transfer knowledge between the two modalities. However, in these methods, knowledge transfer is usually based on the abundance of available data, and the less common case of transferring knowledge from compressed data domain to pixel domain is ignored. This is a shortcoming that limits the use of compressed domain knowledge and transformation of this knowledge into better performance for models in the pixel domain. Therefore, it is necessary to develop a system that will improve model performance in data scarcity situations by transferring compressed domain knowledge into pixel domain and address the shortcomings of current techniques.

[0007] Summary and Objects of the Invention

[0008] The invention relates to a system for improving the performance of computer vision models in cases of data scarcity by transferring knowledge from compressed domain (HEVC) data to the pixel domain by cross-distillation.An object of the invention is to train the student model that will work in the pixel domain using the knowledge of the teacher model trained in the compressed domain, thereby enabling the model to perform better than standard training.

[0009] Another object of the invention is to improve the performance of computer vision models in domains where data is scarce.

[0010] Another object of the invention is to transfer knowledge from compressed domain (HEVC) data to pixel domain.

[0011] A further object of the invention is to enable knowledge transfer between different modalities by cross-distillation.

[0012] Another object of the invention is to improve the training of the student model by using the labeled data as well as the feature maps (hints) obtained from the intermediate layers of the teacher model.

[0013] Another object of the invention is to minimize performance loss due to limited data in pixel domain.

[0014] Another object of the invention is to reduce the impact of missing data due to privacy, cost, or data limitations.

[0015] Description of the Drawings

[0016] Figure 1. A drawing showing a schematic view of the system of the invention.

[0017] Figure 2. A drawing showing the flow diagram of the method of the invention.

[0018] Description of the References in the Drawings

[0019] 1 . Corresponding compressed domain image

[0020] 2. Backbone component of the teacher model

[0021] 3. Neck component of the teacher model

[0022] 4. Head component of the teacher model

[0023] 5. Teacher model6. Pixel domain image

[0024] 7. Student model

[0025] 8. Backbone component of the student model

[0026] 9. Neck component of the student model

[0027] 10. Head component of the student model

[0028] 11. Label

[0029] 12. Knowledge distillation module

[0030] 1001. Extracting a pixel-domain image (6) of the pixel domain from the dataset, 1002. Transmitting the pixel domain image (6) to the student model (7) and generating features for this image with the student model (7).

[0031] 1003. Extracting the corresponding compressed domain image (1) from the dataset 1004. Transmitting corresponding compressed domain image (1) to the teacher model (5) and generating features for this image with the teacher model (5)

[0032] 1005. Receiving label (11 ) data for the input images

[0033] 1006. transferring the knowledge of the teacher model (5) to the student model (7) by the knowledge distillation module (12) and training the student model (7) with the label (11 ) and the features produced by the teacher model (5)

[0034] Detailed Description of the Invention

[0035] The invention relates to a system for improving the performance of computer vision models in cases of data scarcity by transferring knowledge from compressed domain (HEVC) data to the pixel domain by cross-distillation.

[0036] The invention ensures that the student model that will work in the pixel domain is trained using the knowledge of the teacher model trained in the compressed domain, thereby enabling the model to perform better than standard training. Thus, the invention consists of two main stages. In the first stage, the teacher model is trained using compressed domain images and labels. Then, in the second stage, the student model that will work in the pixel domain is trained. From this stage onwards, the teacher model only operates in the inference phase and the knowledge of the teacher model trained in the first phase is used to train the student model.

[0037] The invention solves the problem of data scarcity by enabling the transfer of knowledge from compressed domain (HEVC) data to pixel domain. It uses a "cross-distillation"algorithm to perform this transfer. Cross-distillation refers to the transfer of knowledge between models operating in different modalities. In this approach, which is a type of knowledge distillation method, the model from which knowledge is transferred is called the teacher and the model to which knowledge is transmitted is called the student. In the invention, the knowledge obtained by the teacher model from the compressed data is transferred to the student model to improve its performance in the pixel domain. The proposed approach uses hint transfer, which is a special case of knowledge distillation, i.e., the features obtained in the intermediate layers of the teacher model are transferred to the student model. In this concept, the transferred feature maps are called hints. The invention transfers hints between the backbone components; the first part of the teacher and the student model. Thus, using the knowledge of the teacher model in addition to the label data allows the student model to be trained better and perform better compared to the standard training using only pixel domain data. Thus, the performance loss caused by the lack of data in the pixel domain that the model will process in is prevented.

[0038] The corresponding compressed domain image (1) is a gray-scale image obtained by the estimation-based method proposed in (Beratoglu and Tdreyin, 2021).

[0039] The pixel domain image (6) is an image where each pixel is represented by red, green, and blue (RGB) channel values.

[0040] The teacher model (5) is composed of the backbone component (2) of the teacher model, the neck component (3) of the teacher model, and the head component (4) of the teacher model, and processes the corresponding image.

[0041] The backbone component (2) of the teacher model processes the Corresponding compressed domain image (1 ) to extract features through the operations in the backbone layers.

[0042] These features are used for training the student model with the knowledge distillation module.

[0043] These features are also transmitted to the neck component (3) of the teacher model.The neck component (3) of the teacher model enables the extraction of features through the operations in the layers it contains. These features are transmitted to the head component (4) of the teacher model.

[0044] In the head component (4) of the teacher model, the teacher model processing the corresponding image produces its prediction of the image in the head component. This prediction is class prediction if the problem on which the model is run is classification, or value prediction if it is regression.

[0045] The student model (7) consists of the backbone component (8) of the student model, the neck component (9) of the student model, and the head component (10) of the student model, and processes the pixel domain image (6).

[0046] The backbone component (8) of the student model processing the pixel domain image extracts student model features and attempts to make these features similar to the features obtained by the backbone component of the teacher model with the knowledge distillation module (12). These features are transmitted to the neck component (9) of the student model.

[0047] The neck component (9) of the student model generates features in the neck component of the student model processing the pixel domain image and transmits it to the head component (10) of the student model.

[0048] The head component (10) of the student model generates the prediction of the image.

[0049] Label (11) is the actual correspondence of the instances in the dataset, indicating the data desired for the model to predict. The label information of the images is used for training the student model (7).

[0050] The knowledge distillation module (12) enables the transfer of knowledge from the teacher model (5) to the student model (7). It is used to ensure that the two models produce the same features for corresponding images.

[0051] The backbone component (2) of the teacher model, the neck component (3) of the teacher model, the head component (5) of the teacher model, the teacher model, thestudent model (7), the backbone component (8) of the student model, the neck component (9) of the student model, the head component (10) of the student model, the knowledge distillation module (12) operate on a server. The corresponding compressed domain image (1), pixel domain image (6), and label (11) are stored in databases.

[0052] The process steps of the method the operating the system, running on the server are as follows:

[0053] - Extracting a pixel-domain image (6) of the pixel domain from the dataset (1001 ), - Transmitting the pixel domain image (6) to the student model (7) and generating features for this image with the student model (7) (1002),

[0054] - Extracting the corresponding compressed domain image (1) from the dataset (1003),

[0055] - Transmitting corresponding compressed domain image (1) to the teacher model (5) and generating features for this image with the teacher model (5) (1004), - Receiving label (11 ) data for the input images (1005),

[0056] - transferring the knowledge of the teacher model (5) to the student model (7) by the knowledge distillation module (12) and training the student model (7) with the label (11 ) and the features produced by the teacher model (5) (1006).

[0057] The object of the invention is to train the student model that will work in the pixel domain using the knowledge of the teacher model trained in the compressed domain, thereby enabling the model to perform better than standard training. For this purpose, firstly, the teacher model is trained with compressed domain images using labels (11). Then, the training phase of the student model that will work in the pixel domain begins. From this stage onwards, the teacher model operates only in the inference phase. For the purpose of training the student model, the corresponding compressed-domain version of the image (1 ), which serves as the input to the student model, is provided as input to the pretrained teacher model. The teacher model (5) processing the corresponding image first extracts the features of the image in the backbone component (2). These features are used to train the student model (7) with the knowledge distillation module (12). Furthermore, the features obtained in the backbone component are transmitted to the neck component (3) and then to the head component (4). Then, the teacher model head component (4) processing the corresponding image generates its prediction of the image. Thus, the teacher model consists of three parts.On the other hand, the pixel domain image (6) is transmitted to the student model (7). The student model, like the teacher model, consists of three parts: backbone (8), neck (9), and head (10). The features obtained in the backbone component are transmitted to the neck component (9) and then to the head component (10). In this component, the student model generates its prediction of the image. The training of the student model is based on two key components. The loss function representing these components can be expressed as in Equation (1):

[0058]

[0059] Equation 1

[0060] Wherein A is the equilibrium coefficient between the loss functions, LtaSk is the task loss function and Lhint is the distillation loss function. One of the two components in the training, denoted LtaSk in Equation (1), represents the standard supervised learning approach, which uses label knowledge (11) regarding the images to correct the predictions of the model. The other component is the use of the knowledge distillation module (12), which is the most important part of the invention, to attempt to make the features produced by the student model similar to the features produced by the teacher model. This component is expressed as Lhint in Equation (1) and can be calculated using Equation (2):

[0061] ^hintyLfeal ( Jraw)?)

[0062]

[0063] Equation 2

[0064] Wherein, (lraw, Icomp) e P. P represents the paired dataset containing pixel domain images and their correlating / corresponding compressed domain images, k denotes the number of hints used, i ' is the representation of the ithhint layer of the student model, which maps a raw domain image into a feature map of size (H x w x C)i. <(>' is a function that maps a compressed domain image into a feature map of size (H x w x C)i. H, W and C represent the dimensions of height, width, and number of channels of the generated feature map, respectively. Furthermore, DA and Lfeatrefer to the domain adapter moduleand the feature loss function determined by the hint type, respectively. The domain adapter used to reduce the domain difference between the teacher and student representation matrices consists of a convolution layer.

Claims

CLAIMS1. A model optimization system based on cross-distillation for data scarcity, characterized in that it comprises:- a teacher model (5) operating on a server, comprising at least one backbone component (2) of the teacher model that processes the corresponding compressed domain image (1), extracts features in the backbone component and transmits these features to the neck component (3) of the teacher model; at least one neck component (3) of the teacher model that extracts features in the neck component and transmits these features to the head component (4) of the teacher model; at least one head component (4) of the teacher model that produces the prediction for the image,a student model (7) operating on a server, comprising at least one neck component (8) of the student model that extracts features by processing the pixel domain image (6), makes these features similar to the features of the teacher model with the knowledge distillation module (12), and transmits these features to the neck component (9) of the student model; at least one neck component (9) of the student model that generates features in the student model neck component by processing the pixel domain image (6) and transmits these to the head component (10) of the student model; at least one head component (10) of the student model that produces the prediction for the image- a knowledge distillation module (12) operating on a server, transferring the knowledge of the teacher model (5) to the student model (7) so that the two models produce the same features for the corresponding images.

2. A model optimization method based on cross-distillation for data scarcity, characterized in that it comprises the following process steps operating on a server:extracting a pixel-domain image (6) of the pixel domain from the dataset (1001 ), - transmitting the pixel domain image (6) to the student model (7) and generating features for this image with the student model (7) (1002),extracting the corresponding compressed domain image (1) from the dataset (1003),- transmitting corresponding compressed domain image (1) to the teacher model (5) and generating features for this image with the teacher model (5) (1004), receiving label (11 ) data for the input images (1005),- transferring the knowledge of the teacher model (5) to the student model (7) by the knowledge distillation module (12) and training the student model (7) with the label (11 ) and the features produced by the teacher model (5) (1006).

3. A model optimization system based on cross-distillation for data scarcity according to claim 1 , characterized in that it comprises:a knowledge distillation module (12) calculating the similarity of the features produced by the student model to the features produced by the teacher modelusing the equation4. A model optimization system based on cross-distillation for data scarcity according to claim 1 , characterized in that it comprises the head component (4) of the teacher model that performs class prediction if the problem on which the model is run is classification and value prediction if it is regression.

Images