Knowledge distillation-based model training method and electronic device

Abstract

The present application discloses a knowledge distillation-based model training method and an electronic device. The method comprises: performing feature extraction on a first sample image by means of a plurality of teacher models to obtain foreground supervised sub-features and background supervised sub-features of N local category objects; performing feature extraction on the first sample image by means of a first student model to obtain first foreground features and first background features of the N local category objects; performing foreground knowledge distillation on the first student model on the basis of the foreground supervised sub-features and the first foreground features, and performing background knowledge distillation on the first student model on the basis of the background supervised sub-features and the first background features, wherein the first student model having undergone the foreground knowledge distillation and the background knowledge distillation is a second student model; and performing image prediction on a second sample image by means of the second student model, and adjusting model parameters of the second student model on the basis of a prediction result of the image prediction.

Description

Model training method based on knowledge distillation and electronic device

[0001] Cross-reference

[0002] The present application claims priority to the Chinese patent application No. 202510027202.3, filed on January 8, 2025, and entitled "Model training method based on knowledge distillation and electronic device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of computer vision, and in particular to a model training method based on knowledge distillation and an electronic device. BACKGROUND

[0004] In the industry landing application of AI algorithms, small models have strong concurrent ability and low resource occupation, and are suitable for real-time video streaming, but the generalization ability of small models is weak, and a large number of private industry samples need to be collected for model training, resulting in high data collection cost. In contrast, large models have weak concurrent ability and high resource occupation, but large models have the advantages of strong generalization ability and high precision, and through the help of knowledge distillation technology, the migration of large model knowledge to small models can be completed at low cost, thereby improving the precision of small models under the premise of limited industry samples. Therefore, how to improve the effect of migrating large model knowledge to small models has become one of the problems to be solved in the industry. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a model training method based on knowledge distillation and an electronic device.

[0006] In one aspect, the embodiments of the present application provide a model training method based on knowledge distillation, comprising: extracting features of a first sample image through a plurality of teacher models to obtain foreground supervision sub-features and background supervision sub-features of N local class objects, N being an integer greater than 1; extracting features of the first sample image through a first student model to obtain first foreground features and first background features of the N local class objects; performing foreground knowledge distillation on the first student model according to N foreground supervision sub-features and the first foreground features, and performing background knowledge distillation on the first student model according to the background supervision sub-features and the first background features; the first student model after the foreground knowledge distillation and the background knowledge distillation is a second student model; performing image prediction on a second sample image through the second student model, and adjusting model parameters of the second student model according to a prediction result of the image prediction.

[0007] In another aspect, an embodiment of the present application provides a model training device based on knowledge distillation, comprising: a first extraction module configured to extract features of a first sample image by a plurality of teacher models to obtain foreground supervision sub-features and background supervision sub-features of N local class objects, N being an integer greater than 1; a second extraction module configured to extract features of the first sample image by a first student model to obtain first foreground features and first background features of the N local class objects; a knowledge distillation module configured to perform foreground knowledge distillation on the first student model according to the N foreground supervision sub-features and the first foreground features, and perform background knowledge distillation on the first student model according to the background supervision sub-features and the first background features; the first student model after the foreground knowledge distillation and the background knowledge distillation being a second student model; and an adjustment module configured to perform image prediction on a second sample image by the second student model, and adjust model parameters of the second student model according to a prediction result of the image prediction.

[0008] In another aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being configured to call and execute the computer program from the memory to implement the model training method based on knowledge distillation.

[0009] In another aspect, an embodiment of the present application provides a computer readable storage medium configured to store a computer program, the computer program being executable by a processor to implement the model training method based on knowledge distillation.

[0010] In another aspect, an embodiment of the present application provides a computer program product, comprising a computer program executable by a processor to implement the model training method based on knowledge distillation. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of one or more embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in one or more embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] FIG. 1 is a schematic flowchart of a model training method based on knowledge distillation according to an embodiment of the present application;

[0013] FIG. 2 is a schematic application scenario diagram of a model training method based on knowledge distillation according to an embodiment of the present application;

[0014] Figure 3 is a schematic scene diagram of a model training method based on knowledge distillation according to an embodiment of this application;

[0015] Figure 4 is a schematic diagram illustrating the effect of deleting redundant sub-features according to an embodiment of this application;

[0016] Figure 5 is a schematic diagram illustrating the effect of adding missing sub-features according to an embodiment of this application;

[0017] Figure 6 is a schematic scene diagram of a model training method based on knowledge distillation according to another embodiment of this application;

[0018] Figure 7 is a schematic diagram of a model training method based on knowledge distillation according to an embodiment of this application;

[0019] Figure 8 is a schematic block diagram of a model training device based on knowledge distillation according to an embodiment of this application;

[0020] Figure 9 is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0021] This application provides a model training method and electronic device based on knowledge distillation to solve the technical problem that the transferred model has poor performance in the scenario of model transfer based on knowledge distillation.

[0022] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0023] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein.

[0024] The knowledge distillation-based model training method provided in this application can be executed by an electronic device or by software installed in an electronic device. Specifically, the electronic device can be a terminal device or a server device. The terminal device can include smartphones, laptops, smart wearable devices, vehicle terminals, etc., while the server device can include an independent physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing.

[0025] Figure 1 is a schematic flowchart of a model training method based on knowledge distillation according to an embodiment of this application. As shown in Figure 1, the method includes:

[0026] Step S102: Extract features from the first sample image using multiple teacher models to obtain N foreground-supervised sub-features and background-supervised sub-features for local category objects.

[0027] Where N is an integer greater than 1. The first sample image includes multiple local category objects, which refer to a subset of the multiple category objects in the first sample image. For example, if the first sample image includes people and vehicles, then people and vehicles are respectively local category objects in the first sample image. The foreground supervised sub-features of the local category objects are the foreground features of the local category objects, and the background supervised sub-features of the local category objects are the background features of the local category objects, both of which play a supervisory role in the knowledge distillation process.

[0028] The number of teacher models is matched to the number of local category objects (i.e., the value of N). Optionally, the number of teacher models is N, with each teacher model used to extract foreground supervised sub-features and background supervised sub-features for one local category object. For example, in the first sample image, there are two local category objects: people and vehicles. Teacher model 1 can identify people but not vehicles. Teacher model 2 can identify vehicles but not people. Therefore, during knowledge distillation, both teacher model 1 and teacher model 2 need to be used simultaneously, i.e., the number of teacher models is 2.

[0029] Optionally, the number of teacher models is less than N. One teacher model can be used to extract foreground and background supervised sub-features for one or more local category objects. For example, the first sample image includes three local category objects: people, vehicles, and pets. Teacher model 1 can recognize people but not vehicles or pets. Teacher model 2 can recognize vehicles but not people or pets. Teacher model 3 can recognize vehicles and pets but not people. Therefore, during knowledge distillation, teacher model 1 and teacher model 3 can be used simultaneously. Of course, teacher model 1, teacher model 2, and teacher model 3 can also be used simultaneously, as long as the local category objects recognized by the teacher models can cover all the local category objects required for model training.

[0030] Multiple teacher models can be cross-industry models; for example, multiple teacher models could include a campus teacher model and a transportation teacher model. The campus teacher model can be used to extract people from sample images, while the transportation teacher model can be used to extract vehicles. This allows the student model to learn cross-industry knowledge, concentrating object recognition capabilities from multiple different industries into a single model, greatly improving the model's distillation performance.

[0031] Step S104: Extract features from the first sample image using the first student model to obtain the first foreground features and first background features of N local category objects.

[0032] Among them, the N local category objects extracted by the first student model and the teacher model are the same.

[0033] Step S106: Perform foreground knowledge distillation on the first student model based on N foreground supervised sub-features and the first foreground feature; and perform background knowledge distillation on the first student model based on the background supervised sub-features and the first background feature; the first student model after foreground knowledge distillation and background knowledge distillation is the second student model.

[0034] In this process, foreground knowledge distillation and background knowledge distillation are performed simultaneously. The second student model refers to the intermediate model obtained after knowledge distillation (including foreground knowledge distillation and background knowledge distillation) of the first student model. At this point, it is not the final student model that is expected, and the model parameters of the second student model need to be further optimized.

[0035] In supervised learning, background features complement foreground features and play a crucial role in reducing false positives and improving model generalization. Teacher models inherently possess strong generalization capabilities; in scenarios with limited sample distillation, background feature distillation of the teacher model can transfer its generalization ability to the student model.

[0036] Step S108: Perform image prediction on the second sample image using the second student model, and adjust the model parameters of the second student model based on the prediction results.

[0037] Optionally, the image prediction result includes a foreground prediction result, i.e., the predicted location of the local category object in the second sample image. Step S108 can be performed as follows:

[0038] First, the second student model is used to predict the foreground of the second sample image, resulting in a foreground prediction result containing N local class objects. The second sample image and the first sample image can be the same or different. Optionally, the pre-collected sample images are divided into a training set and a test set at a specific ratio, with the sample images in the training set serving as the first sample image and the sample images in the test set serving as the second sample image.

[0039] Secondly, the model parameters of the second student model are adjusted based on the foreground prediction results and the label information of the second sample image.

[0040] Figure 2 is a schematic application scenario diagram of a model training method based on knowledge distillation according to an embodiment of this application. In this application scenario, N teacher models are used to perform knowledge distillation on a student model. The N teacher models include teacher model 1, teacher model 2, ..., teacher model N. Due to space limitations, Figure 2 only shows teacher model 1 and teacher model N as an example. As shown in Figure 2, a first sample image is simultaneously input to the N teacher models and the first student model. The N teacher models extract features from the N local category objects in the first sample image to obtain teacher features corresponding to each teacher model, such as teacher feature 1, teacher feature 2, ..., teacher feature N. These teacher features, when combined, include foreground supervised sub-features and background supervised sub-features of the N local category objects. At the same time, the first student model extracts features from the N local category objects in the first sample image to obtain student features, including the first foreground features and the first background features of the N local category objects. Then, foreground knowledge distillation is performed on the first student model based on the N foreground supervised sub-features and the first foreground features, and background knowledge distillation is performed on the first student model based on the background supervised sub-features and the first background features to obtain the second student model. Then, the second student model is used to predict the second sample image, and the model parameters of the second student model are adjusted based on the prediction results.

[0041] In this embodiment, the teacher model can be a pre-trained open-source large model. During the training of the student model, the parameters of the teacher model can be frozen and used to extract features from each sample image, serving as supervisory information for the student model's feature extraction, enabling the student model to learn comprehensive and in-depth knowledge. Furthermore, during knowledge distillation, the student model uses training image labels to supervise the model's prediction results, thereby solving the loss function and performing gradient backpropagation.

[0042] When collecting sample images in advance, publicly available industry sample images can be collected online. To ensure the diversity of sample types, images with different shooting angles, lighting, and background environments can be collected as sample images. Furthermore, the sample images should contain all local category objects to be identified. For example, if the goal is to train a student model that can identify N local category objects, then the sample images should contain these N local category objects.

[0043] The technical solution of this application involves extracting features from a first sample image using multiple teacher models to obtain N foreground supervised sub-features and background supervised sub-features for local category objects, where N is an integer greater than 1. A first student model is then used to extract features from the first sample image to obtain N first foreground features and first background features for local category objects. Subsequently, foreground knowledge distillation is performed on the first student model based on the N foreground supervised sub-features and the first foreground features, and background knowledge distillation is performed on the first student model based on the background supervised sub-features and the first background features, resulting in a second student model. Image prediction is then performed on the second sample image using the second student model, and the model parameters of the second student model are adjusted based on the prediction results. It is evident that when performing model transfer based on knowledge distillation, foreground knowledge distillation and background knowledge distillation can be performed simultaneously. This allows the student model to learn not only foreground knowledge from the sample image but also low-heat background knowledge during the knowledge distillation process, thus helping to improve the generalization ability of the student model. Furthermore, since the knowledge distillation process can extract features of multiple local category objects in the sample image, including foreground supervision sub-features, background supervision sub-features, first foreground features, and first background features, and perform knowledge distillation based on the features of multiple local category objects, the student model can learn knowledge of multiple categories at the same time. In other words, a student model can learn cross-category cross-knowledge, thereby enabling the distilled student model to have the ability to recognize different categories of objects. This achieves the effect of distilling out a student model with strong concurrency and high accuracy under the premise of a limited number of samples.

[0044] Figure 3 is a schematic scene diagram of a model training method based on knowledge distillation according to an embodiment of this application. In this embodiment, taking N=2 as an example, the pre-constructed knowledge distillation network architecture includes teacher model 1, teacher model 2, and student model. Teacher model 1 provides vehicle detection capabilities in the transportation domain, i.e., the ability to identify vehicles. Teacher model 2 provides human detection capabilities for park management, i.e., the ability to identify people. As shown in Figure 3, the first sample image includes people and vehicles, and the label information of the first sample image includes the position coordinates of people and vehicles. The first sample image is input into teacher model 1, teacher model 2, and student model respectively. Teacher model 1 extracts the foreground supervised sub-features of vehicles, and teacher model 2 extracts the foreground supervised sub-features of people. The student model extracts the first foreground features of people and vehicles. Based on the foreground supervised sub-features of vehicles and people, knowledge distillation is performed on the student model, ultimately enabling the student model to simultaneously extract people and vehicles. Thus, in the knowledge distillation process, the student model is provided with cross-domain cross-empowerment from the transportation domain to the park management domain, rapidly generating a high-precision, high-concurrency student model with a small number of samples.

[0045] In one embodiment, before executing step S102, multiple teacher models are pre-configured, and a teacher model matching the training scenario of the student model can be selected from these pre-configured teacher models. Optionally, the model category corresponding to the pre-configured teacher model is compared with the label information of the first sample image to obtain a first comparison result. Based on the first comparison result, multiple teacher models matching N local category objects are selected from the pre-configured teacher models. The pre-configured teacher model for each model category is used to identify at least one local category object.

[0046] The label information of the first sample image includes the positional features of N local category objects within the first sample image. These positional features can be represented as positional coordinates. By adding label information to the first sample image, the location of the local category objects within the image can be clearly determined. Optionally, the positional features of the local category objects in the first sample image are represented as the positional coordinates of four key points (i.e., vertices) of a bounding box. The bounding box can be a line frame containing the local category objects and closely approximating their outlines. Of course, the local category objects in this application are not limited to the form of a bounding box; they can also be any other type of shape, such as a circle, square, ellipse, etc.

[0047] Multiple pre-configured teacher models can be large open-source models from different or the same industries. These pre-configured teacher models should, as far as possible, cover the recognition capabilities of various object types to ensure object diversity. The labeling information for the first sample image can be generated using any existing labeling software (e.g., labelImg).

[0048] There are several ways to annotate the label information in the first sample image. Optionally, N local category objects can be labeled in the form of rectangular boxes in the first sample image. Alternatively, the position coordinates of each local category object can be associated with the first sample image as a set, which includes the position coordinates of the four vertices of the rectangular box corresponding to the local category object.

[0049] For example, in the label information, local category objects are represented by positional coordinates [Xmin, Ymin, Xmax, Ymax]. Xmin and Xmax are used to determine the projection position of the bounding box on the horizontal axis of the coordinate system, and Ymin and Ymax are used to determine the projection position of the bounding box on the vertical axis of the coordinate system. Based on the label information, it is possible to map the corresponding feature regions of local category objects in the feature images at each level, thereby finding and extracting the foreground features of each local category object according to the mapped coordinates.

[0050] In this embodiment, by selecting multiple teacher models that match N local category objects from the pre-configured teacher models, the selected multiple teacher models can accurately extract the foreground supervision sub-features of N local category objects, thereby providing accurate supervision information for the knowledge distillation of the student model.

[0051] In one embodiment, when feature extraction is performed on a first sample image using multiple teacher models to obtain foreground supervised sub-features and background supervised sub-features of N local category objects, the following method can be used: if the N local category objects include local category objects that match the model category of the teacher model, feature extraction is performed on the matching local category objects using the teacher model to obtain the foreground supervised sub-features of the matching local category objects.

[0052] The model category of the teacher model is determined based on the local category objects that the teacher model can recognize. For example, if teacher model 1 can recognize local category object A, then teacher model 1 can be considered to have model category A. Local category objects that match the model category of the teacher model refer to the local category objects that the teacher model can recognize. For example, if teacher model 1 can recognize local category object A, then teacher model 1 matches local category object A.

[0053] In one embodiment, feature extraction of a first sample image using multiple teacher models can be performed as follows: Feature extraction is performed on local category objects in the first sample image that match the model categories of the teacher models, resulting in M ​​foreground supervision sub-features for these local category objects, where M is an integer greater than 1. If M is not equal to N, feature alignment processing is performed on the M foreground supervision sub-features based on the label information of the first sample image, resulting in N foreground supervision sub-features. The label information of the first sample image includes the positional features of the N local category objects in the first sample image. That is, the label information of the first sample image is used to indicate the N local category objects.

[0054] In this embodiment, the number of local category objects covered by multiple teacher models may differ from the number of local category objects included in the first sample image. The number of local category objects covered by multiple teacher models refers to the number of local category objects that the multiple teacher models can identify from the first sample image. Based on the label information of the first sample image, the local category objects to be identified can be determined. Therefore, by performing feature alignment processing on the M foreground supervision sub-features using the label information of the first sample image, the foreground supervision sub-features corresponding to the N local category objects can be obtained.

[0055] The feature alignment processing method may include: deleting some foreground supervision sub-features, or adding some foreground supervision sub-features. When M is greater than N, the feature alignment processing method is to delete some foreground supervision sub-features. When M is less than N, the feature alignment processing method is to add some foreground supervision sub-features.

[0056] Optionally, when M is greater than N, redundant sub-features among the M foreground supervision sub-features are determined according to the M foreground supervision sub-features and the label information of the first sample image; the redundant sub-features are deleted from the M foreground supervision sub-features to obtain N foreground supervision sub-features.

[0057] Among them, by comparing the local category objects corresponding to the M foreground supervision sub-features with the N local category objects indicated by the label information of the first sample image, the redundant sub-features among the M foreground supervision sub-features can be determined. For example, multiple teacher models extract foreground supervision sub-features of local category objects of person, vehicle, and pet, that is, M = 3. The label information of the first sample image indicates two local category objects of person and vehicle, that is, N = 2. By comparing the local category objects corresponding to the three foreground supervision sub-features extracted by the multiple teacher models with the two local category objects indicated by the label information of the first sample image, it can be determined that the redundant sub-feature is the foreground supervision sub-feature of the local category object "pet".

[0058] In addition to the foreground supervision sub-features of redundant local category objects, the redundant sub-features may also include the foreground supervision sub-features of overlapping local category objects. For example, teacher model 1 and teacher model 2 have the ability to extract features of the same local category object, and both extract the foreground supervision sub-feature of this local category object during the distillation process. Then, during the feature alignment processing, one of the foreground supervision sub-features can be deleted to avoid the foreground supervision sub-feature of the same local category object being calculated multiple times.

[0059] FIG. 4 is a schematic effect diagram of deleting redundant sub-features according to an embodiment of the present application. As shown in FIG. 4, the first sample image includes a person and a vehicle, and multiple teacher models extract the foreground supervision sub-features of the person and the vehicle. Assume that the label information of the first sample image only includes the position coordinates of the person, that is, the features to be extracted are only the features of the local category object "person". Therefore, the foreground supervision sub-features extracted by the multiple teacher models include redundant sub-features, that is, the features of the local category object "vehicle". By deleting the foreground supervision sub-feature of the local category object "vehicle", the feature alignment processing can be achieved, ensuring that the student model learns the object features consistent with the label information, that is, only learns the detection ability of the person.

[0060] When M is less than N, determine the missing sub-features among the N foreground supervised sub-features according to the M foreground supervised sub-features and the label information of the first sample image; extract the features of the first sample image through a teacher model that matches the missing sub-features to obtain the missing sub-features.

[0061] Among them, by comparing the local category objects corresponding to the M foreground supervised sub-features with the N local category objects indicated by the label information of the first sample image, the missing sub-features among the M foreground supervised sub-features can be determined. For example, multiple teacher models extract the foreground supervised sub-features of two local category objects, vehicle and pet, that is, M = 2. The label information of the first sample image indicates three local category objects, person, vehicle, and pet, that is, N = 3. By comparing the local category objects corresponding to the two foreground supervised sub-features extracted by multiple teacher models with the three local category objects indicated by the label information of the first sample image, it can be determined that the missing sub-feature is the foreground supervised sub-feature of the local category object "person".

[0062] When extracting the features of the first sample image through a teacher model that matches the missing sub-features to obtain the missing sub-features, if the pre-constructed knowledge distillation network framework includes a teacher model that matches the missing sub-features, use this teacher model to extract the features of the first sample image to obtain the missing sub-features. If the pre-constructed knowledge distillation network framework does not include a teacher model that matches the missing sub-features, supervised learning can be performed according to the label information of the first sample image, so as to improve the missing foreground supervised sub-features through multiple rounds of iteration.

[0063] Figure 5 is a schematic effect diagram of adding missing sub-features according to an embodiment of the present application. As shown in Figure 5, the first sample image includes a person and a vehicle, and multiple teacher models only extract the foreground supervised sub-feature of the person. Assume that the label information of the first sample image includes the position coordinates of the person and the vehicle, that is, the features to be extracted are expected to include the features of the local category object "person" and the features of the local category object "vehicle". Therefore, there are missing sub-features among the foreground supervised sub-features extracted by multiple teacher models, that is, the features of the local category object "vehicle". By adding the foreground supervised sub-feature of the local category object "vehicle", feature alignment processing can be achieved, ensuring that the student model learns object features consistent with the label information, that is, learning the detection capabilities of both people and vehicles at the same time.

[0064] In this embodiment, whether there are redundant or missing foreground supervised sub-features extracted by multiple teacher models, the M foreground supervised sub-features can be aligned to N foreground supervised sub-features through feature alignment processing, so as to ensure that the student model can learn the feature information of N local category objects during knowledge distillation, which is beneficial to distilling a student model with strong concurrency ability and high accuracy.

[0065] In one embodiment, when multiple teacher models are used to extract features from the first sample image to obtain foreground supervision sub-features and background supervision sub-features of N local category objects, the following steps A1 - A2 can be performed:

[0066] Step A1: Use the teacher model to extract features from the first sample image to obtain foreground supervision sub-features of local category objects that match the model category of the teacher model.

[0067] Step A2: Remove the first image region corresponding to the foreground supervision sub-features and the second image region corresponding to the redundant foreground features from the first sample image to obtain the background supervision sub-features. The second image region is the image region corresponding to the local category objects that the teacher model did not extract.

[0068] In this embodiment, for each teacher model, the foreground supervision sub-features and background supervision sub-features of the corresponding local category objects can be obtained by performing steps A1 - A2.

[0069] FIG. 6 is a schematic scenario diagram of a model training method based on knowledge distillation according to another embodiment of the present application. Compared with the embodiment shown in FIG. 3, FIG. 6 adds a background knowledge distillation module. The distilled background feature region is the image region obtained after removing the first image region corresponding to the foreground supervision sub-features and the second image region corresponding to the redundant foreground features. The first image region corresponding to the foreground supervision sub-features is the image region where the local category objects that match the model category of the teacher model are located. The second image region corresponding to the redundant foreground features is the image region where the local category objects that do not match the model category of the teacher model are located. Optionally, if the first sample image also includes local category objects that do not match the label information, that is, local category objects that are not expected to be extracted, then when determining the background feature region, the image region where the local category objects that are not expected to be extracted are located needs to be removed. That is to say, the image region where the local category objects that are not expected to be extracted are located is also considered as the background feature region.

[0070] For example, the first sample image includes a person and a vehicle. The teacher model 1 provides the vehicle detection ability in the transportation field, that is, the ability to recognize vehicles. Then, the local category object matching the model category of the teacher model 1 is the vehicle, and the local category object not matching the model category of the teacher model 1 is the person. When determining the background supervision sub-feature corresponding to the teacher model 1, the first image region corresponding to the foreground supervision sub-feature, that is, the image region where the vehicle is located, is removed from the first sample image, and the second image region corresponding to the redundant foreground feature, that is, the image region where the person is located, is removed, and then the background feature region can be obtained. The teacher model extracts features from the background feature region to obtain the background supervision sub-feature corresponding to the teacher model. Each teacher model extracts the background supervision sub-feature in the same way, and thus N background supervision sub-features are obtained.

[0071] As shown in FIG. 6, the teacher model 1 extracts the foreground supervision sub-feature and the background supervision sub-feature of the vehicle, the teacher model 2 extracts the foreground supervision sub-feature and the background supervision sub-feature of the person, and the student model extracts the first foreground feature and the first background feature of the vehicle and the person. All the foreground features and background features extracted by the teacher model 1 and the teacher model 2 are used as the supervision information of the student model, so as to realize the synchronous progress of foreground feature distillation and background feature distillation. It can be seen that when the teacher model 1, the teacher model 2 and the student model extract the background features, the corresponding background feature regions are the same, and they are all the image regions obtained after removing the full foreground feature region from the first sample image. The full foreground feature region is the image region where all the local category objects to be extracted are located.

[0072] In one embodiment, when performing foreground knowledge distillation on the first student model according to the N foreground supervision sub-features and the first foreground feature, first, the N foreground supervision sub-features are concatenated to obtain the foreground supervision feature of N local category objects; secondly, according to the difference degree between the foreground supervision feature and the first foreground feature, the distillation foreground loss function of the first student model is calculated, and the model parameters of the first student model are adjusted according to the distillation foreground loss function.

[0073] Optionally, the foreground distillation loss function can be expressed as the following formula:

[0074] The pixel value of a certain pixel point in the extracted first foreground feature, C represents the number of channels of the foreground supervision feature and the first foreground feature. n represents the feature level of the feature extraction network in the teacher model and the student model.

[0075] Similarly, when performing background knowledge distillation on the first student model based on the background supervision sub-features and the first background feature, first, the N background supervision sub-features are concatenated to obtain the background supervision features of N local class objects; second, according to the difference degree between the background supervision features and the first background feature, the distillation background loss function of the first student model is calculated, and the model parameters of the first student model are adjusted according to the distillation background loss function.

[0076] The representation of the background distillation loss function and the foreground distillation loss function is similar, as shown in the following formula (2):

[0077] The pixel value of a certain pixel point in the extracted first background feature, C represents the number of channels of the background supervision feature and the first background feature. n represents the feature level of the feature extraction network in the teacher model and the student model.

[0078] In one embodiment, when performing foreground knowledge distillation on the first student model according to N foreground supervision sub-features and the first foreground feature, the following steps B1-B3 are executed:

[0079] Step B1, for each foreground supervision sub-feature, determine the first foreground sub-feature in the first foreground feature that matches the foreground supervision sub-feature.

[0080] Step B2, according to the difference degree between the foreground supervision sub-feature and the first foreground sub-feature, calculate the local foreground loss function of the first student model. Since there are N foreground supervision sub-features, N local foreground loss functions can be obtained through calculation.

[0081] Step B3, according to the N local foreground loss functions, determine the distillation foreground loss function of the first student model, and adjust the model parameters of the first student model according to the distillation foreground loss function.

[0082] Optionally, the distillation foreground loss function of the first student model is equal to the sum of the N local foreground loss functions. When executing step B2, each local foreground loss function can be calculated in the manner of the above formula (1).

[0083] When performing background knowledge distillation on the first student model according to the background supervision sub-features and the first background feature, the following steps C1-C3 are executed:

[0084] Step C1, for each background supervision sub-feature, determine the first background sub-feature in the first background feature that matches the background supervision sub-feature.

[0085] Step C2: Calculate the local background loss function of the first student model according to the difference degree between the background supervision sub-features and the first background sub-features. Since there are N background supervision sub-features, N local background loss functions can be obtained through calculation.

[0086] Step C3: Determine the distilled background loss function of the first student model according to the N local background loss functions, and adjust the model parameters of the first student model according to the distilled background loss function.

[0087] Optionally, the distilled background loss function of the first student model is equal to the sum of the N local background loss functions. When performing Step C2, each local background loss function can be calculated in the manner of the above formula (2).

[0088] In this embodiment, by calculating the local foreground loss function and the local background loss function corresponding to each local category object respectively, and then determining the total distilled foreground loss function according to the N local foreground loss functions, and determining the total distilled background loss function according to the N local background loss functions, the effective calculation of the loss function in the scenario of distilling one student model by using multiple teacher models is realized. After calculating the distilled foreground loss function and the distilled background loss function, gradient backpropagation can be performed based on the distilled foreground loss function and the distilled background loss function to achieve knowledge transfer from multiple teacher models to one student model.

[0089] In one embodiment, the prediction result of the image prediction includes a foreground prediction result, and the foreground prediction result includes: the predicted position information of N local category objects in the second sample image. The label information of the second sample image includes: the reference position information of N local category objects in the second sample image. The reference position information is the correct position information of the local category object in the second sample image and can be represented as position coordinates.

[0090] When adjusting the model parameters of the second student model according to the foreground prediction result and the label information of the second sample image, the supervision loss function of the second sample image can be determined first according to the difference degree between the predicted position information and the reference position information. Then, the model parameters of the second student model are adjusted according to the supervision loss function.

[0091] In one embodiment, since the foreground feature and the background feature are separated, there may be an imbalance between the foreground feature and the background feature. To alleviate this imbalance problem, it can be eliminated by preprocessing the global features (including foreground features and background features) of the first sample image. Optionally, by guiding the student model to capture and enhance the relationship between image channels and the positional relationship between pixel points, and performing knowledge distillation on the enhanced feature image. Specifically, it includes the following steps D1 - D4:

[0092] Step D1: Determine the global supervision features of multiple teacher models based on N foreground supervision sub-features and background supervision sub-features. Moreover, determine the global student features of the second student model based on the first foreground feature and the first background feature.

[0093] Step D2: Perform feature enhancement processing on the global supervision features to obtain enhanced global supervision features.

[0094] Step D3: Perform feature enhancement processing on the global student features to obtain enhanced global student features.

[0095] Step D2 and Step D3 can be executed synchronously, and there is no sequential execution relationship.

[0096] Step D4: Adjust the model parameters of the second student model according to the difference degree between the enhanced global supervision features and the enhanced global student features.

[0097] Taking the feature enhancement processing of the global supervision features as an example. First, perform feature compression on the global supervision features in a specified dimension to obtain the global supervision compressed features in the specified dimension. Assume the global supervision feature is a 2D feature F C ∈R C×H×W , perform feature mean pooling in the H and W directions respectively to obtain the coordinate features in the H and W directions, which are used to capture the long-range dependencies in the spatial direction and the position information of the space respectively, and perform feature splicing, so as to obtain the global supervision compressed feature with the dimension of C×1×(W + H). The acquisition method of the global supervision compressed feature F S can be expressed by the following formula (3):

[0098] where i and j respectively represent the pixel values of a certain pixel point on the feature image in the H and W directions.

[0099] Secondly, perform feature activation on the global supervision compressed features to obtain the activated global supervision compressed features. Based on the specified dimension, perform compression restoration on the activated global supervision compressed features to obtain the activated global supervision features.

[0100] The purpose of feature activation is to reduce the complexity of the network and improve the generalization ability of the model. An optional activation method is: first, use a fully connected layer to compress the channel dimension of the global supervision feature F S with a compression ratio of r to obtain the compressed feature Then perform batch normalization and feature activation on the feature to reduce model overfitting and introduce non-linearity, and then complete it through a fully connected layer restoration in the channel dimension at a ratio of 1 / r to obtain the activated global supervision feature FE The process of the feature activation network can be expressed by the following formula:

[0101] Where, FC(F S ) represents processing the global supervision feature F S through a fully connected layer. represents batch normalization of the feature obtained after FC(F S ). ReLU represents feature activation.

[0102] Again, use the activated global supervision feature for feature enhancement processing to obtain an enhanced global supervision feature.

[0103] Optionally, the feature enhancement processing method is: weighting the activated global supervision feature with the original feature (i.e., the global supervision feature) to achieve feature enhancement of the channel coordinates. First, split the activated global supervision feature F E in the H and W directions to obtain feature weights and and normalize the feature weights to enable them to weight the original feature, as shown in the following formula:

[0104] Where, F O is the enhanced global supervision feature.

[0105] The process of feature enhancement processing for the global student feature is: performing feature compression on the global student feature in a specified dimension to obtain a globally compressed student feature in the specified dimension; performing feature activation on the globally compressed student feature to obtain an activated globally compressed student feature; based on the specified dimension, performing compression restoration on the activated globally compressed student feature to obtain an activated global student feature; using the activated global student feature for feature enhancement processing to obtain an enhanced global student feature. The specific process of feature enhancement processing for the global student feature is the same as that of the global supervision feature and will not be repeated here.

[0106] The finally output enhanced global supervision feature and global student feature are enhanced in terms of channels and coordinates, and the loss function is solved by the loss function solving method provided in the foregoing embodiments. By performing feature enhancement processing on the global supervision feature and the global student feature, the feature imbalance problem caused by stripping foreground features and background features during the knowledge distillation process can be eliminated, and the distillation effect of the model can be improved.

[0107] Next, a specific embodiment is used to illustrate the model training method based on knowledge distillation provided in this application.

[0108] First, the implementation principle of the model training method based on knowledge distillation will be described. FIG. 7 is a schematic principle diagram of a model training method based on knowledge distillation according to an embodiment of the present application. As shown in FIG. 7, a network model architecture is pre-built, including multiple teacher models and one student model. Due to space limitations, FIG. 7 only shows one teacher model and one student model. The teacher model includes a backbone network and a feature pyramid, and the student model includes a lightweight backbone network, a feature pyramid, and a network head. The sample images are respectively input into the teacher model and the student model, and the foreground supervision sub-features and background supervision sub-features of N local category objects are extracted by the teacher model. The foreground supervision sub-features of the N local category objects are concatenated to obtain the foreground supervision feature. The background supervision sub-features of the N local category objects are concatenated to obtain the background supervision feature. The foreground features and background features of the N local category objects are extracted by the student model. The feature levels of the features extracted by the teacher model and the student model are the same. Then, the foreground supervision sub-features, background supervision sub-features, foreground features, and background features are input into the multi-teacher feature distillation module for knowledge distillation, so that the student model can learn the recognition capabilities of multiple teacher models for the N local category objects. The network head connected to the feature pyramid of the student model is used to output the prediction result of the student model for the sample image, and the prediction result includes the position prediction of the N local category objects. Then, the loss function is calculated according to the prediction result and the label information of the sample image, and the model parameters of the student model are adjusted according to the loss function, so as to obtain the trained student model.

[0109] Based on the network model architecture shown in FIG. 7, the training of the port monitoring model will be described below as an example.

[0110] First, sample images are obtained. The port includes scenarios of personnel control and dangerous operations, and the local category objects to be detected include personnel, vehicles, and operation equipment. The offline video in this scenario is collected, and the frame-extracted images of the offline video are obtained, and the valid images are selected from them as sample images. The valid images refer to the images whose clarity meets the preset requirements and contain at least one local category object to be detected. In addition, publicly available data samples can be obtained as a supplement to the sample images. The collected image set can be divided into a training set and a test set according to a certain ratio. Among them, the training set is used for knowledge distillation, and the test set is used for performance testing and optimization of the distilled second student model. After obtaining the sample images, the label information of the sample images needs to be labeled. The label information includes the position coordinates of the local category objects to be detected in the sample images.

[0111] Secondly, construct a pre-trained large model (i.e., the teacher model) and the student model. Since the local category objects to be detected include personnel, vehicles, and operation equipment, multiple teacher models need to be pre-configured to enable the multiple teacher models to have the recognition ability of personnel, vehicles, and operation equipment. In this embodiment, 2 teacher models are configured, including a human detection model and a vehicle equipment detection model. The human detection model is used to identify personnel in the sample image, and the vehicle equipment detection model is used to identify vehicles and operation equipment in the sample image. Initialize the model parameters of the human detection model and the vehicle equipment detection model. In addition, according to the resource conditions of the port, construct a network model structure that meets the on-site real-time performance, including basic structures such as a backbone network, a feature pyramid, and a network head, and initialize the parameters of the network model structure, so as to generate a student model for distillation.

[0112] After that, perform knowledge distillation based on the constructed network model structure. Input the sample images containing label information in the training set into the human detection model, the vehicle equipment detection model, and the student model for feature extraction, and output multi-level feature images through the feature pyramids of each model. For the human detection model and the vehicle equipment detection model, according to the label information, obtain the position coordinates mapped by the label information from the multi-level feature images, so as to separate the foreground supervised sub-features. Then splice the multiple foreground supervised sub-features to obtain the foreground supervised feature. Among them, after obtaining the foreground supervised feature, the image region obtained by removing the image region where all the foreground supervised features in the sample image are located is the background feature region. In this embodiment, the image region remaining after removing the regions where personnel, vehicles, and operation equipment are located in the sample image is the background image region, and the background supervised feature can be obtained by extracting the image features on the background image region.

[0113] During knowledge distillation, the distillation of the foreground supervised feature and the background supervised feature is carried out synchronously. For the foreground supervised feature, first perform feature alignment processing on the foreground supervised feature and the foreground feature extracted by the student model. The feature alignment processing methods may include: deleting some foreground supervised sub-features, or adding some foreground supervised sub-features. The specific feature alignment processing methods have been described in detail in the above embodiments and will not be elaborated here. After that, use the MSE loss function shown in formula (1) as the foreground distillation loss function, and perform loss calculation and gradient backpropagation on the aligned teacher model and student model to achieve the foreground knowledge distillation of the multi-teacher model. For the background supervised feature, the MSE loss function shown in formula (2) can be used as the background distillation loss function, and perform loss calculation and gradient backpropagation on the aligned teacher model and student model to achieve the background knowledge distillation of the multi-teacher model.

[0114] Before loss solving, feature enhancement processing can be performed on the features respectively extracted by the teacher model and the student model. Optionally, obtain the full-scale features of the human detection model, the vehicle device detection model, and the student model at a certain feature layer, and then perform feature enhancement on the obtained full-scale features. The specific feature enhancement method has been described in detail in the above embodiments and will not be elaborated here. When solving the loss, according to the prediction result output by the network head of the student model and the label information of the sample image, loss solving and gradient backpropagation are performed.

[0115] It can be seen that by adopting the technical solution provided by the embodiment of the present application, when performing model migration based on knowledge distillation, foreground knowledge distillation and background knowledge distillation can be carried out synchronously, so that the student model can not only learn the foreground knowledge in the sample image but also learn the low-heat background knowledge during the knowledge distillation process. Therefore, it helps to improve the generalization ability of the student model. In addition, since in the knowledge distillation process, the features of multiple local category objects in the sample image can be extracted, including the foreground supervision sub-features, background supervision sub-features, foreground features, and background features of personnel, vehicles, and operation equipment, and knowledge distillation is carried out based on the features of multiple local category objects, the student model can learn multiple categories of knowledge simultaneously. That is to say, a student model can learn cross-category cross knowledge, so that the distilled student model has the ability to recognize different category objects, achieving the effect of distilling a student model with strong concurrency ability and high accuracy under the premise of a limited number of samples.

[0116] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing may be advantageous.

[0117] The above is the model training method based on knowledge distillation provided by the embodiment of the present application. Based on the same idea, the embodiment of the present application also provides a model training device based on knowledge distillation.

[0118] FIG. 8 is a schematic block diagram of a model training device based on knowledge distillation according to an embodiment of the present application. As shown in FIG. 8, the device includes:

[0119] The first extraction module 81 is configured to extract features from the first sample image through multiple teacher models to obtain foreground supervision sub-features and background supervision sub-features of N local category objects, where N is an integer greater than 1; the second extraction module 82 is configured to extract features from the first sample image through the first student model to obtain the first foreground feature and the first background feature of the N local category objects; the knowledge distillation module 83 is configured to perform foreground knowledge distillation on the first student model according to the N foreground supervision sub-features and the first foreground feature, and perform background knowledge distillation on the first student model according to the background supervision sub-features and the first background feature; the first student model after the foreground knowledge distillation and the background knowledge distillation is the second student model; the adjustment module 84 is configured to perform image prediction on the second sample image through the second student model, and adjust the model parameters of the second student model according to the prediction result of the image prediction.

[0120] In one embodiment, the apparatus further includes: a comparison module, configured to compare the model category corresponding to the pre-configured teacher model with the label information of the first sample image before extracting the foreground supervision sub-features and the background supervision sub-features of the N local category objects from the first sample image through the multiple teacher models, to obtain a first comparison result; each pre-configured teacher model of each model category is used to identify at least one local category object; the label information of the first sample image includes: the position features of the N local category objects in the first sample image; a screening module, configured to screen out the multiple teacher models that match the N local category objects from the pre-configured teacher models according to the first comparison result.

[0121] In one embodiment, when the first extraction module 81 extracts the foreground supervision sub-features and the background supervision sub-features of the N local category objects from the first sample image through the multiple teacher models, the following steps are performed: in the case that the N local category objects include local category objects that match the model category of the teacher model, extract features from the matching local category objects through the teacher model to obtain the foreground supervision sub-features of the matching local category objects.

[0122] In one embodiment, when the first extraction module 81 extracts features from the first sample image through multiple teacher models to obtain foreground supervised sub-features and background supervised sub-features of N local category objects, the following steps are performed: Extract features from local category objects in the first sample image that match the model category of the teacher model to obtain M foreground supervised sub-features of local category objects; M is an integer greater than 1; In the case where M is not equal to N, perform feature alignment processing on the M foreground supervised sub-features according to the label information of the first sample image to obtain the N foreground supervised sub-features; The label information of the first sample image includes: the position features of the N local category objects in the first sample image.

[0123] In one embodiment, when the first extraction module 81 performs feature alignment processing on the M foreground supervised sub-features according to the label information of the first sample image to obtain the N foreground supervised sub-features in the case where M is not equal to N, the following steps are performed: In the case where M is greater than N, determine redundant sub-features in the M foreground supervised sub-features according to the M foreground supervised sub-features and the label information of the first sample image; Delete the redundant sub-features from the M foreground supervised sub-features to obtain the N foreground supervised sub-features; In the case where M is less than N, determine missing sub-features in the N foreground supervised sub-features according to the M foreground supervised sub-features and the label information of the first sample image; Extract features from the first sample image through a teacher model that matches the missing sub-feature to obtain the missing sub-feature.

[0124] In one embodiment, when the first extraction module 81 extracts features from the first sample image through multiple teacher models to obtain foreground supervised sub-features and background supervised sub-features of N local category objects, the following steps are performed: Extract features from the first sample image through the teacher model to obtain foreground supervised sub-features of local category objects that match the model category of the teacher model; Remove the first image area corresponding to the foreground supervised sub-feature and the second image area corresponding to the redundant foreground feature from the first sample image to obtain the background supervised sub-feature; The second image area is the image area corresponding to the local category object that the teacher model did not extract.

[0125] In one embodiment, when the knowledge distillation module 83 performs foreground knowledge distillation on the first student model according to N foreground supervision sub-features and the first foreground feature, the following steps are executed: concatenate the N foreground supervision sub-features to obtain the foreground supervision feature of the N local class objects; calculate the distillation foreground loss function of the first student model according to the difference degree between the foreground supervision feature and the first foreground feature, and adjust the model parameters of the first student model according to the distillation foreground loss function.

[0126] In one embodiment, when the knowledge distillation module 83 performs foreground knowledge distillation on the first student model according to N foreground supervision sub-features and the first foreground feature, the following steps are executed: for each foreground supervision sub-feature, determine the first foreground sub-feature in the first foreground feature that matches the foreground supervision sub-feature; calculate the local foreground loss function of the first student model according to the difference degree between the foreground supervision sub-feature and the first foreground sub-feature; determine the distillation foreground loss function of the first student model according to N local foreground loss functions, and adjust the model parameters of the first student model according to the distillation foreground loss function.

[0127] In one embodiment, the prediction result of the image prediction includes: a foreground prediction result; when the adjustment module 84 performs image prediction on a second sample image through the second student model and adjusts the model parameters of the second student model according to the prediction result of the image prediction, the following steps are executed: perform foreground prediction on the second sample image through the second student model to obtain the foreground prediction result including the N local class objects; adjust the model parameters of the second student model according to the foreground prediction result and the label information of the second sample image.

[0128] In one embodiment, the foreground prediction result includes: the predicted position information of the N local class objects in the second sample image; the label information of the second sample image includes: the reference position information of the N local class objects in the second sample image; when the adjustment module 84 adjusts the model parameters of the second student model according to the foreground prediction result and the label information of the second sample image, the following steps are executed: determine the supervision loss function of the second sample image according to the difference degree between the predicted position information and the reference position information; adjust the model parameters of the second student model according to the supervision loss function.

[0129] In one embodiment, the apparatus further includes: a determination module, configured to determine global supervision features of the plurality of teacher models according to the N foreground supervision sub-features and the background supervision sub-feature; determine global student features of the second student model according to the first foreground feature and the first background feature; a feature enhancement module, configured to perform feature enhancement processing on the global supervision features to obtain enhanced global supervision features; perform feature enhancement processing on the global student features to obtain enhanced global student features; a second adjustment module, configured to adjust model parameters of the second student model according to a difference degree between the enhanced global supervision features and the enhanced global student features.

[0130] By using the apparatus of the embodiment of the present application, the first sample image is subjected to feature extraction by a plurality of teacher models to obtain N foreground supervision sub-features and a background supervision sub-feature of local category objects, where N is an integer greater than 1; the first sample image is subjected to feature extraction by a first student model to obtain a first foreground feature and a first background feature of the N local category objects. Furthermore, foreground knowledge distillation is performed on the first student model according to the N foreground supervision sub-features and the first foreground feature, and background knowledge distillation is performed on the first student model according to the background supervision sub-feature and the first background feature to obtain a second student model, and the second student model is used to perform image prediction on a second sample image, and the model parameters of the second student model are adjusted according to the prediction result of the image prediction. It can be seen that when performing model transfer based on knowledge distillation, foreground knowledge distillation and background knowledge distillation can be performed synchronously, so that the student model can not only learn foreground knowledge in the sample image but also learn low-heat background knowledge during the knowledge distillation process, which helps to improve the generalization ability of the student model. In addition, since during the knowledge distillation process, features of multiple local category objects in the sample image can be extracted, including foreground supervision sub-features, background supervision sub-features, first foreground features, and first background features, and knowledge distillation is performed based on the features of the multiple local category objects, the student model can learn multiple categories of knowledge at the same time. That is to say, a student model can learn cross-category cross knowledge, so that the distilled student model has the ability to recognize different category objects, achieving the effect of distilling a student model with strong concurrency ability and high accuracy under the premise of a limited number of samples.

[0131] Those skilled in the art should understand that the model training apparatus based on knowledge distillation in FIG. 8 can be used to implement the model training method based on knowledge distillation described above, and the detailed description thereof should be similar to the description in the method part above. To avoid redundancy, it will not be described in detail here.

[0132] Based on the same idea, an embodiment of the present application further provides an electronic device, as shown in FIG. 9. Electronic devices can vary greatly due to configuration or performance differences, and may include one or more processors 901 and a memory 902. One or more application programs or data may be stored in the memory 902. Among them, the memory 902 can be short-term storage or persistent storage. The application programs stored in the memory 902 may include one or more modules (not shown in the figure), and each module may include a series of computer-executable instructions for the electronic device. Further, the processor 901 can be set to communicate with the memory 902 and execute a series of computer-executable instructions in the memory 902 on the electronic device. The electronic device may also include one or more power supplies 903, one or more wired or wireless network interfaces 904, one or more input / output interfaces 905, and one or more keyboards 906.

[0133] Specifically, in this embodiment, the electronic device includes a memory and one or more programs, where one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the electronic device, and is configured to be executed by one or more processors. The one or more programs include the following computer-executable instructions: extracting foreground supervision sub-features and background supervision sub-features of N local category objects from a first sample image through multiple teacher models, where N is an integer greater than 1; extracting a first foreground feature and a first background feature of the N local category objects from the first sample image through a first student model; performing foreground knowledge distillation on the first student model according to the N foreground supervision sub-features and the first foreground feature, and performing background knowledge distillation on the first student model according to the background supervision sub-features and the first background feature; the first student model after the foreground knowledge distillation and the background knowledge distillation is the second student model; performing image prediction on a second sample image through the second student model, and adjusting the model parameters of the second student model according to the prediction result of the image prediction.

[0134] Adopting the technical solution of the embodiment of the present application, multiple teacher models are used to extract features from the first sample image, obtaining foreground supervision sub-features and background supervision sub-features of N local category objects, where N is an integer greater than 1; the first student model is used to extract features from the first sample image, obtaining the first foreground feature and the first background feature of the N local category objects. Then, foreground knowledge distillation is performed on the first student model according to the N foreground supervision sub-features and the first foreground feature, and background knowledge distillation is performed on the first student model according to the background supervision sub-features and the first background feature, obtaining a second student model, and the second student model is used to perform image prediction on the second sample image, and the model parameters of the second student model are adjusted according to the prediction result of the image prediction. It can be seen that when performing model transfer based on knowledge distillation, foreground knowledge distillation and background knowledge distillation can be carried out synchronously, enabling the student model to not only learn the foreground knowledge in the sample image but also learn the low-heat background knowledge during the knowledge distillation process, thus helping to improve the generalization ability of the student model. In addition, since during the knowledge distillation process, features of multiple local category objects in the sample image can be extracted, including foreground supervision sub-features, background supervision sub-features, the first foreground feature, and the first background feature, and knowledge distillation is performed based on the features of multiple local category objects, the student model can learn various types of knowledge simultaneously. That is to say, a student model can learn cross-category cross knowledge, so that the distilled student model has the ability to recognize different category objects, achieving the effect of distilling a student model with strong concurrent ability and high accuracy under the premise of a limited number of samples.

[0135] The embodiment of the present application also proposes a computer-readable storage medium that stores one or more computer programs. The one or more computer programs include instructions that, when executed by an electronic device including multiple application programs, can enable the electronic device to execute each process of the above-mentioned embodiment of the model training method based on knowledge distillation, and are specifically used to execute: using multiple teacher models to extract features from the first sample image, obtaining foreground supervision sub-features and background supervision sub-features of N local category objects, where N is an integer greater than 1; using the first student model to extract features from the first sample image, obtaining the first foreground feature and the first background feature of the N local category objects; performing foreground knowledge distillation on the first student model according to the N foreground supervision sub-features and the first foreground feature, and performing background knowledge distillation on the first student model according to the background supervision sub-features and the first background feature; the first student model after the foreground knowledge distillation and the background knowledge distillation is the second student model; using the second student model to perform image prediction on the second sample image, and adjusting the model parameters of the second student model according to the prediction result of the image prediction.

[0136] Adopting the technical solution of the embodiment of the present application, multiple teacher models are used to extract features from the first sample image, obtaining foreground supervision sub-features and background supervision sub-features of N local category objects, where N is an integer greater than 1; the first student model is used to extract features from the first sample image, obtaining the first foreground feature and the first background feature of N local category objects. Furthermore, foreground knowledge distillation is performed on the first student model according to the N foreground supervision sub-features and the first foreground feature, and background knowledge distillation is performed on the first student model according to the background supervision sub-features and the first background feature, obtaining a second student model, and the second student model is used to perform image prediction on the second sample image, and the model parameters of the second student model are adjusted according to the prediction result of the image prediction. It can be seen that when performing model transfer based on knowledge distillation, foreground knowledge distillation and background knowledge distillation can be carried out simultaneously, enabling the student model to not only learn the foreground knowledge in the sample image but also learn the low-heat background knowledge during the knowledge distillation process, thus helping to improve the generalization ability of the student model. In addition, since during the knowledge distillation process, features of multiple local category objects in the sample image can be extracted, including foreground supervision sub-features, background supervision sub-features, the first foreground feature and the first background feature, and knowledge distillation is performed based on the features of multiple local category objects, the student model can learn multiple categories of knowledge simultaneously. That is to say, a student model can learn cross-category cross knowledge, so that the distilled student model has the ability to recognize different category objects, achieving the effect of distilling a student model with strong concurrent ability and high accuracy under the premise of a limited number of samples.

[0137] The embodiment of the present application provides a computer program product, including a computer program, which is executed by a processor to implement each process of the above embodiment of the model training method based on knowledge distillation and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0138] The system, device, module or unit illustrated in the above embodiment can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0139] For the convenience of description, the above device is described by dividing it into various units according to functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0140] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0144] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0145] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0146] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0147] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0148] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0149] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0150] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the scope of the claims of this application.

Claims

1. A model training method based on knowledge distillation, comprising: By extracting features from the first sample image using multiple teacher models, N foreground-supervised sub-features and background-supervised sub-features of local class objects are obtained, where N is an integer greater than 1; The first sample image is used to extract features using the first student model to obtain the first foreground features and the first background features of the N local category objects; Based on N foreground supervised sub-features and the first foreground feature, foreground knowledge distillation is performed on the first student model, and background knowledge distillation is performed on the first student model based on the background supervised sub-features and the first background feature; the first student model after the foreground knowledge distillation and the background knowledge distillation is the second student model. The second student model is used to predict the second sample image, and the model parameters of the second student model are adjusted based on the prediction results.

2. The method according to claim 1, wherein, Before extracting features from the first sample image using multiple teacher models to obtain foreground-supervised sub-features and background-supervised sub-features of N local category objects, the method further includes: The model category corresponding to the pre-configured teacher model is compared with the label information of the first sample image to obtain a first comparison result; the pre-configured teacher model of each model category is used to identify at least one local category object; the label information of the first sample image includes: the positional features of the N local category objects in the first sample image; Based on the first comparison result, the plurality of teacher models that match the N local category objects are selected from the pre-configured teacher models.

3. The method according to claim 1, wherein, The process involves extracting features from the first sample image using multiple teacher models to obtain N foreground-supervised sub-features and background-supervised sub-features for local class objects, including: If the N local category objects include local category objects that match the model category of the teacher model, feature extraction is performed on the matching local category objects by the teacher model to obtain the foreground supervision sub-features of the matching local category objects.

4. The method according to claim 1, wherein, The process involves extracting features from the first sample image using multiple teacher models to obtain N foreground-supervised sub-features and background-supervised sub-features for local class objects, including: Feature extraction is performed on local category objects in the first sample image that match the model category of the teacher model to obtain M foreground supervision sub-features of local category objects; M is an integer greater than 1; When M is not equal to N, feature alignment processing is performed on the M foreground supervision sub-features according to the label information of the first sample image to obtain the N foreground supervision sub-features; the label information of the first sample image includes: the position features of the N local category objects in the first sample image.

5. The method according to claim 4, wherein, When M is not equal to N, feature alignment processing is performed on the M foreground supervision sub-features based on the label information of the first sample image to obtain the N foreground supervision sub-features, including: When M is greater than N, based on the M foreground supervision sub-features and the label information of the first sample image, redundant sub-features are determined among the M foreground supervision sub-features; the redundant sub-features are deleted from the M foreground supervision sub-features to obtain the N foreground supervision sub-features. When M is less than N, based on the M foreground supervision sub-features and the label information of the first sample image, the missing sub-features among the N foreground supervision sub-features are determined; the missing sub-features are obtained by extracting features from the first sample image using a teacher model that matches the missing sub-features.

6. The method according to claim 1, wherein, The process involves extracting features from the first sample image using multiple teacher models to obtain N foreground-supervised sub-features and background-supervised sub-features for local class objects, including: The teacher model is used to extract features from the first sample image to obtain foreground supervision sub-features of local category objects that match the model category of the teacher model; The background supervision sub-feature is obtained by removing the first image region corresponding to the foreground supervision sub-feature and the second image region corresponding to the redundant foreground feature from the first sample image; the second image region is the image region corresponding to the local category object that was not extracted by the teacher model.

7. The method according to claim 1, wherein, The step of performing foreground knowledge distillation on the first student model based on N foreground supervised sub-features and the first foreground feature includes: The N foreground supervision sub-features are concatenated to obtain the foreground supervision features of the N local category objects; Based on the difference between the foreground supervision feature and the first foreground feature, the distillation foreground loss function of the first student model is calculated, and the model parameters of the first student model are adjusted according to the distillation foreground loss function.

8. The method according to claim 1, wherein, The step of performing foreground knowledge distillation on the first student model based on N foreground supervised sub-features and the first foreground feature includes: For each foreground supervised sub-feature, determine the first foreground sub-feature that matches the foreground supervised sub-feature in the first foreground features; The local foreground loss function of the first student model is calculated based on the difference between the foreground supervised sub-feature and the first foreground sub-feature. Based on the N local foreground loss functions, the distillation foreground loss function of the first student model is determined, and the model parameters of the first student model are adjusted according to the distillation foreground loss function.

9. The method according to claim 1, wherein, The prediction results of the image prediction include: foreground prediction results; The step of performing image prediction on the second sample image using the second student model, and adjusting the model parameters of the second student model based on the prediction results, includes: The foreground prediction of the second sample image is performed by the second student model to obtain the foreground prediction result containing the N local category objects; Based on the foreground prediction results and the label information of the second sample image, the model parameters of the second student model are adjusted.

10. The method according to claim 9, wherein, The foreground prediction result includes: the predicted position information of the N local category objects in the second sample image; the label information of the second sample image includes: the reference position information of the N local category objects in the second sample image; The step of adjusting the model parameters of the second student model based on the foreground prediction result and the label information of the second sample image includes: The supervised loss function of the second sample image is determined based on the degree of difference between the predicted location information and the reference location information. The model parameters of the second student model are adjusted according to the supervised loss function.

11. The method according to claim 7, wherein, The method further includes: Based on the N foreground supervision sub-features and the background supervision sub-features, the global supervision features of the multiple teacher models are determined; based on the first foreground feature and the first background feature, the global student features of the second student model are determined. The global supervision features are subjected to feature enhancement processing to obtain enhanced global supervision features; The global student features are subjected to feature enhancement processing to obtain enhanced global student features; The model parameters of the second student model are adjusted based on the degree of difference between the enhanced global supervision features and the enhanced global student features.

12. An electronic device comprising a processor and a memory electrically connected to the processor, the memory storing a computer program, the processor being configured to call and execute the computer program from the memory to implement the knowledge distillation-based model training method as described in any one of claims 1-11.

13. A computer-readable storage medium for storing a computer program that can be executed by a processor to implement the knowledge distillation-based model training method as described in any one of claims 1-11.

14. A computer program product comprising a computer program executed by a processor to implement the knowledge distillation-based model training method as described in any one of claims 1-11.

Images