Data labeling method and apparatus, electronic device and storage medium
By using a semi-automated data annotation method that combines annotation models with manual annotation, the problem of low data annotation efficiency in existing technologies is solved, and efficient and accurate data annotation and model training are achieved.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- NANJING YIMU INTELLIGENT TECHNOLOGY CO LTD
- Filing Date
- 2025-06-04
- Publication Date
- 2026-06-04
AI Technical Summary
In existing technologies, data annotation mainly relies on manual labor, which leads to low efficiency and consumes a lot of time and energy.
A semi-automated data annotation method is adopted, which uses an annotation model to infer the data to be annotated, annotates the data based on the inference results, and automatically identifies and stores the annotation difference data through training the model, and optimizes the optimal label by combining manual annotation.
It reduces manual annotation work, improves the efficiency of data annotation, optimizes the model training process, and improves the accuracy and efficiency of data annotation.
Smart Images
Figure CN2025099131_04062026_PF_FP_ABST
Abstract
Description
A data annotation method, apparatus, electronic device, and storage medium Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a data annotation method, apparatus, electronic device and storage medium. Background Technology
[0002] Data annotation is an important task in the field of artificial intelligence. Data annotation refers to assigning reliable label information to data. For example, in image classification tasks, data annotation can refer to labeling the true categories of training images. These true categories can then be used as supervision signals to supervise the training of image classification models, enabling the models to learn the correspondence between the content of training images and their true categories, and thus effectively classify other images.
[0003] However, current data annotation mainly relies on a large amount of human resources, resulting in very low efficiency and wasting a lot of time and energy. Summary of the Invention
[0004] The main objective of this application is to provide a data annotation method, apparatus, electronic device, and storage medium that can perform effective data annotation and improve the efficiency of data annotation.
[0005] On the one hand, embodiments of this application propose a data annotation method, which includes the following steps:
[0006] Obtain multiple data points to be labeled;
[0007] The annotation model is used to perform inference on each of the unannotated data, and the inference result corresponding to each of the unannotated data is output.
[0008] Based on the inference results corresponding to each of the data to be labeled, data labels are applied to each of the data to be labeled.
[0009] In some embodiments, the step of using a labeling model to infer the inference results for each of the unlabeled data and outputting the inference results corresponding to each of the unlabeled data specifically includes:
[0010] Each of the data to be labeled is input into the labeling model, and the labeling model is used to predict the data to be labeled, and the inference label and inference confidence corresponding to each of the data to be labeled are output.
[0011] In some embodiments, the step of labeling each of the data to be labeled according to the inference result corresponding to each of the data to be labeled specifically includes:
[0012] Based on the inference confidence level corresponding to the data to be labeled, determine whether to label the model to be labeled according to the inference label;
[0013] When the inference confidence is less than a given confidence threshold, the data to be labeled is determined as data with uncertain labeling and stored in a third labeled dataset;
[0014] When the inference confidence is greater than the confidence threshold, the inference label is used to annotate the data to be labeled to obtain the corresponding second labeled data, and the second labeled data is stored in the second labeled dataset.
[0015] In some embodiments, the method further includes:
[0016] Obtain the first labeled dataset; the first labeled dataset includes multiple manually labeled first labeled datasets;
[0017] The annotation model is trained using the first and second annotation datasets to determine multiple annotation difference data and the inference difference results corresponding to each annotation difference data. Each annotation difference data and the corresponding inference difference results are stored in the third annotation dataset. The inference difference results include the previous training label, the current training label, and the current inference confidence corresponding to the current training label of the annotation difference data.
[0018] In some embodiments, training the annotation model using the first annotation dataset and the second annotation dataset, determining multiple annotation difference data and inference difference results corresponding to each annotation difference data, and storing each annotation difference data and the corresponding inference difference results in the third annotation dataset specifically includes:
[0019] Dynamically obtain the current data change ratio, which is the ratio of the amount of data that has changed in the first labeled dataset and the second labeled dataset to the amount of training data in the previous model. The amount of training data in the previous model is the total amount of training data used to train the labeled model in the first labeled dataset and the second labeled dataset during the previous labeled model training.
[0020] When the current data change rate exceeds a preset rate threshold, multiple training data are determined from the first labeled dataset and the second labeled dataset, the training label corresponding to each training data is obtained as the previous training label, each training data is input into the labeling model, and the labeling model is used to output the current training label and the current inference confidence corresponding to each training data.
[0021] When the current training label corresponding to the training data is inconsistent with the previous training label, the training data is determined as the annotation difference data. The annotation difference data is then bound to the previous training label, the current training label, and the current inference confidence, and stored in the third annotation dataset.
[0022] In some embodiments, the method further includes:
[0023] According to a preset data selection strategy, multiple inference difference data are determined from the third labeled dataset;
[0024] In response to the manual annotation operation of each of the inference difference data, the optimal inference label for each of the inference difference data is determined, the inference difference data is annotated using the optimal inference label, the annotated determined data corresponding to each of the inference difference data is obtained, and the annotated determined data is stored in the first annotated dataset.
[0025] In some embodiments, determining multiple inference discrepancy data from the third labeled dataset according to a preset data selection strategy specifically includes:
[0026] Multiple annotation discrepancy data and multiple annotation uncertainty data are selected from the third annotation dataset;
[0027] Based on the inference confidence corresponding to each of the labeled uncertain data, sort them in ascending order, and determine multiple first inference difference data from the multiple labeled uncertain data. The first inference difference data are the labeled uncertain data whose ranking in ascending order is less than a preset first ranking threshold.
[0028] Based on the current inference confidence of each of the labeled difference data, they are sorted in descending order, and multiple second inference difference data are determined from the multiple labeled difference data. The second inference difference data are the labeled difference data whose ranking in descending order is less than a preset second ranking threshold.
[0029] On the other hand, embodiments of this application propose a data annotation device, the device comprising:
[0030] The first module is used to acquire multiple data sets to be labeled.
[0031] The second module is used to perform inference on each of the data to be labeled using the labeling model and output the inference results corresponding to each of the data to be labeled.
[0032] The third module is used to annotate each of the data to be annotated according to the inference results corresponding to each of the data to be annotated.
[0033] On the other hand, embodiments of this application propose an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the data annotation method described above.
[0034] On the other hand, embodiments of this application propose a storage medium storing a computer program that, when executed by a processor, implements the data annotation method described above.
[0035] The embodiments of this application include at least the following beneficial effects: The data annotation method, apparatus, electronic device, and storage medium provided in this application acquire multiple data to be annotated; use an annotation model to infer each data to be annotated, and output the inference result corresponding to each data to be annotated; and annotate each data to be annotated according to the inference result corresponding to each data to be annotated. This application can perform effective data annotation, reduce manual annotation work, and improve the efficiency of data annotation. Attached Figure Description
[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0037] Figure 1 is a flowchart of a data annotation method provided in an embodiment of this application;
[0038] Figure 2 is a schematic diagram of the semi-automatic annotation structure in an embodiment of this application;
[0039] Figure 3 is a schematic diagram of a data annotation device provided in an embodiment of this application;
[0040] Figure 4 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0041] Relying on manual data annotation requires a significant investment of manpower in both data annotation and model training. Data engineers need to manually examine each image and label it manually, using methods such as directly categorizing files by folder or using annotation tools. The workload of manual annotation is enormous. When training models, AI engineers often need to continuously adjust the data and repeat training based on actual conditions to obtain a better model. This process of continuous training, inference, and data adjustment consumes a lot of time and energy.
[0042] Based on this, embodiments of this application propose a data annotation method, apparatus, electronic device, and storage medium, which can achieve data annotation in a semi-automatic manner, reduce excessive involvement of AI engineers, and improve the efficiency of data annotation.
[0043] Referring to Figure 1, which is an optional flowchart of a data annotation method provided in an embodiment of this application, the method may include, but is not limited to, steps S101 to S103:
[0044] Step S101: Obtain multiple data points to be labeled;
[0045] Step S102: Use the annotation model to perform inference on each data to be annotated and output the inference result corresponding to each data to be annotated;
[0046] Step S103: Based on the inference results corresponding to each data to be labeled, perform data labeling on each data to be labeled.
[0047] In some embodiments, the first annotation dataset is used to store multiple first annotation data that have been manually annotated and have high confidence; the second annotation dataset is used to store second annotation data that have been inferred by the model, obtained corresponding inference labels, and have high inference confidence; the third annotation dataset is used to store data waiting for manual annotation, including annotation difference data and inference difference data; and the fourth annotation dataset is used to store multiple data that have not been manually annotated or inferred by the model, i.e., the aforementioned data to be annotated.
[0048] In step S102 of some embodiments, the inference result includes inference label and inference confidence. Specifically, each data to be labeled is input into the labeling model, the labeling model is used to perform data prediction on each data to be labeled, and the inference label and inference confidence corresponding to each data to be labeled are output.
[0049] In step S103 of some embodiments, it is determined whether to label the model to be labeled according to the inference confidence corresponding to the data to be labeled. Optionally, when the inference confidence is less than a given confidence threshold, the data to be labeled is determined as data with uncertain labeling and stored in the third labeling dataset; when the inference confidence is greater than the confidence threshold, the data to be labeled is labeled using the inference label to obtain the corresponding second labeled data and the second labeled data is stored in the second labeling dataset.
[0050] When the inference confidence of the data to be labeled is less than the given confidence threshold, it means that the model cannot accurately determine the inference label of the data to be labeled. Therefore, the data to be labeled is identified as data with uncertain labeling, and then stored in the third labeled dataset for manual labeling.
[0051] In some embodiments, the above data annotation method may also include, but is not limited to, steps S201 to S202:
[0052] Step S201: Obtain the first labeled dataset; the first labeled dataset includes multiple manually labeled first labeled datasets;
[0053] Step S202: Train the annotation model using the first and second annotation datasets, determine multiple annotation difference data and the inference difference results corresponding to each annotation difference data, and store each annotation difference data and the corresponding inference difference results in the third annotation dataset. The inference difference results include the previous training label, the current training label, and the current inference confidence corresponding to the current training label of the annotation difference data.
[0054] In some embodiments, the annotation model is automatically trained using a first annotation dataset and a second annotation dataset to determine multiple annotation difference data. Optionally, the training data is the first annotation data or the second annotation data. The training label of the training data at this time is the previous training label. Then, the training data is input into the annotation model, and the annotation model is used to perform model inference. The inference label and inference confidence of the training data are output. The inference label is determined as the current training label of the training data, and the inference confidence is determined as the current inference confidence of the training data.
[0055] In some embodiments, step S202 may include, but is not limited to, steps S301 to S303:
[0056] Step S301: Dynamically obtain the current data change ratio. The current data change ratio is the ratio of the amount of data that has changed in the first labeled dataset and the second labeled dataset to the amount of training data in the previous model. The amount of training data in the previous model is the total amount of training data used to train the labeled model in the first labeled dataset and the second labeled dataset during the previous labeled model training.
[0057] Step S302: When the current data change rate exceeds the preset rate threshold, multiple training data are determined from the first labeled dataset and the second labeled dataset, the training label corresponding to each training data is obtained as the previous training label, each training data is input into the labeling model, and the labeling model is used to output the current training label and the current inference confidence corresponding to each training data.
[0058] Step S303: When the current training label corresponding to the training data is inconsistent with the previous training label, the training data is determined as labeled difference data. After binding the labeled difference data with the previous training label, the current training label and the current inference confidence, it is stored in the third labeled dataset.
[0059] In some embodiments, data changes may include, but are not limited to, data modification and data addition. For example, when data is added to the first labeled dataset or the second labeled dataset, such as adding multiple first labeled data to the first labeled dataset, the amount of newly added data M1 is recorded. When data is modified in the first labeled dataset or the second labeled dataset, such as changing the inference label of the data, the amount of modified information M2 is recorded. The amount of training data Mc of the previous model and the ratio threshold M are obtained. G When (M1+M2) / Mc>M G At that time, multiple training data are randomly selected from the first and second labeled datasets, and the labeled model is trained again using the multiple training data.
[0060] In some embodiments, when the current training label corresponding to the training data is inconsistent with the previous training label, it indicates that there is a difference between the manual annotation result of the training data and the model inference result, or the model inference results of the two consecutive times are inconsistent. Therefore, the training data needs to be manually annotated again, the training data is identified as the annotation difference data, and then stored in the third annotation dataset.
[0061] In some embodiments, the above data annotation method may also include, but is not limited to, steps S401 to S402:
[0062] Step S401: Based on the preset data selection strategy, determine multiple inference difference data from the third labeled dataset;
[0063] Step S402: In response to the manual annotation operation of each inference difference data, determine the optimal inference label for each inference difference data, use the optimal inference label to annotate the inference difference data, obtain the labeled data corresponding to each inference difference data, and store the labeled data in the first labeled dataset.
[0064] In some embodiments, based on a data selection strategy, multiple inference difference data with significant differences are selected from the third labeled dataset. The inference difference data are labeled difference data or labeled uncertain data. The optimal inference label for the inference difference data is determined by manual labeling. The inference difference data is labeled using the optimal inference label to obtain the corresponding labeled data. The labeled data can be used to train the model more effectively and improve the accuracy of the model's inference.
[0065] In some embodiments, referring to FIG2, FIG2 is an optional structural schematic diagram of semi-automatic annotation in an embodiment of the present application, wherein multiple annotation difference data are determined from a first annotation dataset and a second dataset, multiple annotation uncertainty data are determined from a fourth annotation dataset, and each standard difference data and each annotation uncertainty data are stored in a third annotation dataset. Then, multiple inference difference data are determined from the third annotation dataset, and each inference difference data is manually annotated using a manual annotation method to obtain the corresponding annotation determination data. Then, each annotation determination data is stored in the first annotation dataset.
[0066] In some embodiments, step S401 may include, but is not limited to, steps S501 to S503:
[0067] Step S501: Select multiple annotation discrepancy data and multiple annotation uncertainty data from the third annotation dataset;
[0068] Step S502: Sort the data in ascending order according to the inference confidence corresponding to each labeled uncertain data, and determine multiple first inference difference data from multiple labeled uncertain data. The first inference difference data are the labeled uncertain data whose ranking in ascending order is less than a preset first ranking threshold.
[0069] Step S503: Sort the labeled difference data in descending order according to the current inference confidence of each labeled difference data, and determine multiple second inference difference data from the multiple labeled difference data. The second inference difference data are the labeled difference data whose ranking in descending order is less than the preset second ranking threshold.
[0070] In some embodiments, the data are sorted in ascending order based on the inference confidence corresponding to each labeled uncertain data. That is, they are sorted from low to high based on the inference confidence corresponding to each labeled uncertain data. Based on a preset first ranking threshold, all labeled uncertain data before the first ranking threshold are taken as inference difference data. In this way, labeled uncertain data with low inference confidence can be effectively filtered out.
[0071] In some embodiments, the annotation difference data are sorted in descending order based on the current inference confidence of each annotation difference data, that is, sorted from high to low based on the current inference confidence of each annotation difference data. According to a preset second ranking threshold, all annotation difference data before the second ranking threshold are taken as inference difference data. The higher the current inference confidence of the annotation difference data, the greater the difference between the previous training label and the current training label. In this way, annotation difference data with large training label differences can be effectively filtered out.
[0072] In some embodiments, if the third annotation dataset includes multiple image data, the data selection strategy may also include an image similarity filtering strategy, which calculates the image similarity between each image data and other image data, filters out several target image data from the multiple image data, and determines the target image data as inference difference data if the image similarity between the target image data and other image data is lower than the similarity threshold, and uses manual annotation to annotate the target image data.
[0073] Referring to Figure 3, which is an optional structural schematic diagram of a data annotation device provided in an embodiment of this application, the device is used to implement the above-described data annotation method and may include:
[0074] The first module is used to acquire multiple data sets to be labeled.
[0075] The second module is used to perform inference on each data to be labeled using the labeling model and output the inference results corresponding to each data to be labeled.
[0076] The third module is used to annotate each data point based on the inference results corresponding to each data point.
[0077] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0078] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described data annotation method. This electronic device can be any smart terminal, including a tablet computer.
[0079] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0080] Please refer to Figure 4, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes:
[0081] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.
[0082] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the data annotation method of the embodiments of this application.
[0083] The input / output interface 903 is used to implement information input and output;
[0084] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0085] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);
[0086] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.
[0087] This application also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described data annotation method.
[0088] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0089] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0090] This application provides a data annotation method, apparatus, electronic device, and storage medium, which acquires multiple data sets to be annotated; uses an annotation model to infer the meaning of each data set, outputs the inference result corresponding to each data set; and annotates each data set according to the inference result. This application enables effective data annotation, reduces manual annotation work, and improves the efficiency of data annotation.
[0091] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.
[0092] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0094] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0095] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0096] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0097] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0098] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with a computer program, wherein such a storage medium causes the computer to operate in a specific and predefined manner—according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).
[0099] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0100] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A data annotation method, characterized in that, The method includes the following steps: Obtain multiple data points to be labeled; The annotation model is used to perform inference on each of the unannotated data, and the inference result corresponding to each of the unannotated data is output. Based on the inference results corresponding to each of the data to be labeled, data labels are applied to each of the data to be labeled.
2. The data annotation method according to claim 1, characterized in that, The step of using the annotation model to infer the meaning of each of the unannotated data and outputting the inference result corresponding to each of the unannotated data specifically includes: Each of the data to be labeled is input into the labeling model, and the labeling model is used to predict the data to be labeled, and the inference label and inference confidence corresponding to each of the data to be labeled are output.
3. The data annotation method according to claim 2, characterized in that, The step of labeling each of the data to be labeled according to the inference results corresponding to each of the data to be labeled specifically includes: Based on the inference confidence level corresponding to the data to be labeled, determine whether to label the model to be labeled according to the inference label; When the inference confidence is less than a given confidence threshold, the data to be labeled is determined as data with uncertain labeling and stored in a third labeled dataset; When the inference confidence is greater than the confidence threshold, the inference label is used to annotate the data to be labeled to obtain the corresponding second labeled data, and the second labeled data is stored in the second labeled dataset.
4. The data annotation method according to claim 3, characterized in that, The method further includes: Obtain the first labeled dataset; the first labeled dataset includes multiple manually labeled first labeled datasets; The annotation model is trained using the first and second annotation datasets to determine multiple annotation difference data and the inference difference results corresponding to each annotation difference data. Each annotation difference data and the corresponding inference difference results are stored in the third annotation dataset. The inference difference results include the previous training label, the current training label, and the current inference confidence corresponding to the current training label of the annotation difference data.
5. The data annotation method according to claim 4, characterized in that, The step of training the annotation model using the first and second annotation datasets, determining multiple annotation discrepancies and corresponding inference discrepancies for each annotation discrepancy, and storing each annotation discrepancy and its corresponding inference discrepancy in the third annotation dataset specifically includes: Dynamically obtain the current data change ratio, which is the ratio of the amount of data that has changed in the first labeled dataset and the second labeled dataset to the amount of training data in the previous model. The amount of training data in the previous model is the total amount of training data used to train the labeled model in the first labeled dataset and the second labeled dataset during the previous labeled model training. When the current data change rate exceeds a preset rate threshold, multiple training data are determined from the first labeled dataset and the second labeled dataset, the training label corresponding to each training data is obtained as the previous training label, each training data is input into the labeling model, and the labeling model is used to output the current training label and the current inference confidence corresponding to each training data. When the current training label corresponding to the training data is inconsistent with the previous training label, the training data is determined as the annotation difference data. The annotation difference data is then bound to the previous training label, the current training label, and the current inference confidence, and stored in the third annotation dataset.
6. The data annotation method according to claim 4, characterized in that, The method further includes: According to a preset data selection strategy, multiple inference difference data are determined from the third labeled dataset; In response to the manual annotation operation of each of the inference difference data, the optimal inference label for each of the inference difference data is determined, the inference difference data is annotated using the optimal inference label, the annotated determined data corresponding to each of the inference difference data is obtained, and the annotated determined data is stored in the first annotated dataset.
7. The data annotation method according to claim 6, characterized in that, The step of determining multiple inference discrepancy data from the third labeled dataset according to a preset data selection strategy specifically includes: Multiple annotation discrepancy data and multiple annotation uncertainty data are selected from the third annotation dataset; Based on the inference confidence corresponding to each of the labeled uncertain data, sort them in ascending order, and determine multiple first inference difference data from the multiple labeled uncertain data. The first inference difference data are the labeled uncertain data whose ranking in ascending order is less than a preset first ranking threshold. Based on the current inference confidence of each of the labeled difference data, they are sorted in descending order, and multiple second inference difference data are determined from the multiple labeled difference data. The second inference difference data are the labeled difference data whose ranking in descending order is less than a preset second ranking threshold.
8. A data annotation device, characterized in that, The device includes: The first module is used to acquire multiple data sets to be labeled. The second module is used to perform inference on each of the data to be labeled using the labeling model and output the inference results corresponding to each of the data to be labeled. The third module is used to annotate each of the data to be annotated according to the inference results corresponding to each of the data to be annotated.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the data annotation method according to any one of claims 1 to 7.
10. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the data annotation method according to any one of claims 1 to 7.