Image annotation method and apparatus, model training method and apparatus, and electronic device
By displaying images on electronic devices and receiving user annotations, the model is trained using the annotated images to automatically annotate unannotated images, solving the problem of low image annotation efficiency and achieving efficient target detection model training.
Patent Information
- Application Number
- PCT/CN2026/082540
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2026-03-10
- Publication Date
- 2026-11-12
AI Technical Summary
Current image annotation technologies are inefficient, require a lot of manual work, resulting in high time costs and making it difficult to train object detection models efficiently.
By displaying the image to be labeled on an electronic device, receiving the target bounding boxes and category labels labeled by the user, and automatically labeling the target bounding boxes and category labels of the unlabeled image using the target detection model trained on the labeled image, combined with the intelligent appending function, the amount of manual labeling is reduced.
It improves image annotation efficiency, reduces time costs, and enables more efficient training of object detection models.
Smart Images

Figure CN2026082540_12112026_PF_FP_ABST
Abstract
Description
Image annotation methods, model training methods, devices and electronic equipment
[0001] This application claims priority to Chinese Patent Application No. 202510590431.6, filed on May 8, 2025, entitled "Image Annotation Method, Model Training Method, Apparatus and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of machine vision technology, and in particular to image annotation methods, model training methods, devices and electronic equipment. Background Technology
[0003] For object detection models, image annotation refers to selecting the location of the target object of interest in the image and adding a category label to indicate its category, so that the model can accurately extract the features of the target object when training the object detection model.
[0004] However, the work of annotating images usually needs to be done manually. In the training process of an object detection model, multiple images usually need to be annotated, and each image may contain multiple target objects. Therefore, the amount of data to be annotated is large, which leads to low annotation efficiency and consumes a lot of time. Summary of the Invention
[0005] The purpose of this application is to provide image annotation methods, model training methods, apparatus, and electronic devices to improve the efficiency of image annotation. The specific technical solutions are as follows:
[0006] In a first aspect, embodiments of this application provide an image annotation method applied to an electronic device, the method comprising:
[0007] Display the first training image to be labeled;
[0008] Receive user annotations for the first training image, and display the bounding boxes and category labels of the user-annotated target objects on the first training image; wherein the category labels annotate at least one target object;
[0009] In response to the automatic labeling command, the bounding boxes and category labels of the target objects in the second training image are displayed on the second training image; wherein, the bounding boxes and category labels of the target objects in the second training image are obtained by inputting the second training image into a target detection model trained using the labeled first training image; the labeled first training image includes: user-labeled bounding boxes and category labels.
[0010] Secondly, embodiments of this application provide an image annotation method applied to an electronic device, the method comprising:
[0011] Display a training image that needs to be labeled;
[0012] The system receives user annotations for the training image and displays the bounding boxes and category labels of the user-annotated target objects on the training image; wherein the category labels annotate at least one target object.
[0013] In response to a label append instruction, the bounding box and category label of the target object in the training image are displayed on the training image;
[0014] The bounding boxes and category labels of the target objects in the training images are obtained by inputting the training images into a target detection model trained using user-annotated training images.
[0015] Thirdly, embodiments of this application provide a model training method applied to electronic devices, the method comprising:
[0016] Obtain the training images specified by the user;
[0017] The training images are labeled using any of the image labeling methods described above;
[0018] In response to the model training instruction, the target detection model is trained using the labeled training images to obtain the trained target detection model.
[0019] Fourthly, embodiments of this application provide an image annotation apparatus applied to an electronic device, the apparatus comprising:
[0020] The display module is used to display the first training image to be labeled.
[0021] The annotation receiving module is used to receive user annotations for the first training image and display the bounding boxes and category labels of the user-annotated target objects on the first training image; wherein the category labels annotate at least one target object;
[0022] An automatic labeling response module is used to respond to an automatic labeling command and display the bounding boxes and category labels of the target objects in the second training image on the second training image; wherein the bounding boxes and category labels of the target objects in the second training image are obtained by inputting the second training image into a target detection model trained using a labeled first training image; the labeled first training image includes user-labeled bounding boxes and category labels.
[0023] Fifthly, embodiments of this application provide an image annotation apparatus applied to an electronic device, the apparatus comprising:
[0024] The training image display module is used to display a training image to be labeled;
[0025] The user annotation receiving module is used to receive user annotations for the training image and display the bounding boxes and category labels of the user-annotated target objects on the training image; wherein the category labels annotate at least one target object;
[0026] The label appending instruction response module is used to display on the training image: the bounding box and category label of the target object in the training image; wherein, the bounding box and category label of the target object in the training image are obtained by inputting the training image into a target detection model trained using user-annotated training images.
[0027] Sixthly, embodiments of this application provide a model training apparatus applied to an electronic device, the apparatus comprising:
[0028] The training image acquisition module is used to acquire the training images specified by the user.
[0029] The training image annotation module is used to annotate the training image using any of the image annotation methods described above;
[0030] The model training instruction response module is used to respond to model training instructions and train the object detection model using the labeled training images to obtain the trained object detection model.
[0031] In a seventh aspect, embodiments of this application provide an electronic device, including:
[0032] Memory, used to store computer programs;
[0033] The processor, when executing a program stored in memory, implements any of the above-described image annotation methods or model training methods.
[0034] Eighthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described image annotation methods or model training methods.
[0035] In a ninth aspect, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the image annotation methods or model training methods described above.
[0036] Beneficial effects of the embodiments in this application:
[0037] By applying this embodiment of the invention, a user can first manually annotate the first training image displayed on the electronic device, enabling the electronic device to train an object detection model using the annotated first training image. Then, the user can use an automatic labeling command to have the electronic device input the second training image into the trained object detection model, thereby automatically obtaining the bounding boxes and category labels of the target objects in the second training image. It is evident that by applying this embodiment of the invention, only manual annotation of the first training image is required, while the second training image can be automatically annotated by the electronic device. Therefore, this solution enables more efficient image annotation and reduces annotation time costs. Attached Figure Description
[0038] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.
[0039] Figure 1 is a schematic diagram of the structure of the few-sample target detection training operator in an embodiment of this application;
[0040] Figure 2 is a schematic diagram of the principle of training an object detection model using the few-shot object detection training operator shown in Figure 1;
[0041] Figure 3 is a flowchart illustrating an embodiment of the first image annotation method provided in this application.
[0042] Figure 4 is a flowchart illustrating another embodiment of the first image annotation method provided in this application.
[0043] Figure 5(a) is a first schematic diagram of the interactive interface used by the image annotation method provided in the embodiments of this application (state without imported images);
[0044] Figure 5(b) is a second schematic diagram of the interactive interface shown in Figure 5(a) (displaying the status of commonly used target detection parameters);
[0045] Figure 6 is a third schematic diagram of the interactive interface shown in Figure 5(a) (the state of importing images);
[0046] Figure 7 is a fourth schematic diagram of the interactive interface shown in Figure 5(a) (the state when creating a label);
[0047] Figure 8 is the fifth schematic diagram of the interactive interface shown in Figure 5(a) (the state after the label is created);
[0048] Figure 9 is a sixth schematic diagram of the interactive interface shown in Figure 5(a) (the state after the user annotates the first training image);
[0049] Figure 10 is the seventh schematic diagram of the interactive interface shown in Figure 5(a) (showing the status of the automatic marking button);
[0050] Figure 11 is a schematic diagram of the intelligent addition process in the embodiment shown in Figure 4;
[0051] Figure 12(a) is a schematic diagram of label addition in the image annotation method provided in the embodiment of this application;
[0052] Figure 12(b) is a schematic diagram showing that all target objects are annotated in the image annotation method provided in the embodiment of this application;
[0053] Figure 13 is a schematic diagram of the intelligent addition interaction principle of the embodiment shown in Figure 4;
[0054] Figure 14 is a schematic diagram of the automatic annotation process in the embodiment shown in Figure 4;
[0055] Figure 15 is the eighth schematic diagram of the interactive interface shown in Figure 5(a) (the state of prompting rejection of adding the image to the target training set);
[0056] Figure 16 is the ninth schematic diagram of the interactive interface shown in Figure 5(a) (the state prompting whether to retrain the model);
[0057] Figure 17 is a schematic diagram of the automatic marking interaction principle of the embodiment shown in Figure 4;
[0058] Figure 18 is a flowchart illustrating an embodiment of the second image annotation method provided in this application.
[0059] Figure 19 is a flowchart illustrating another embodiment of the second image annotation method provided in this application.
[0060] Figure 20 is a flowchart illustrating yet another embodiment of the second annotation method for images provided in this application.
[0061] Figure 21 is a flowchart illustrating an embodiment of the model training method provided in this application.
[0062] Figure 22 is a schematic diagram of an embodiment of the image annotation device provided in this application;
[0063] Figure 23 is a schematic diagram of another embodiment of the image annotation device provided in this application;
[0064] Figure 24 is a schematic diagram of the model training device provided in the embodiment of this application;
[0065] Figure 25 is a schematic diagram of the structure of the electronic device provided in the embodiment of this application. Detailed Implementation
[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in this application are within the scope of protection of this application.
[0067] In this embodiment, the target detection model to be trained can be a deep learning model, such as a few-shot object detection model (FSOD). To train this model, a dedicated training program can be developed, which can be called a few-shot object detection training operator. See Figure 1, which is a schematic diagram of the structure of the few-shot object detection training operator in this embodiment. As shown in Figure 1, the program can encapsulate the following: a basic deep learning model for object detection (i.e., the target detection model to be trained, hereinafter referred to as the target detection model), commonly used target detection parameters, and target detection annotation functions. For example, commonly used target detection parameters may include: angle enable, used by the user to determine whether to use a rectangle with an angle value to annotate the target object; optimal model resolution setting enable, used by the user to determine whether to adjust the resolution of the displayed training image; and optimal model resolution, used by the user to set the resolution of the displayed training image.
[0068] Users can configure commonly used parameters for the object detection model through this program, and use the object detection annotation function to mark the target objects of interest in the training images. Finally, the basic object detection deep learning model can be trained based on the annotated training images to obtain an object detection model suitable for the current real-world scenario. Of course, in addition to few-shot object detection models, the object detection model to be trained can also be other models, such as the You Only Look Once (YOLO) model, the Single Shot MultiBox Detector (SSD) model, etc.
[0069] To address the aforementioned object detection annotation function, a method is needed to improve the efficiency of annotating training images, thereby improving the efficiency of training the object detection model. Therefore, this application provides an image annotation method, a model training method, an apparatus, and an electronic device. The image annotation method provided in this application is described below. This method can be applied to an electronic device that integrates a display screen to show the interactive interface described in this embodiment. For example, the electronic device can be a computer, tablet computer, smartphone, etc.
[0070] Referring to Figure 2, which is a schematic diagram illustrating the principle of training an object detection model using the few-shot object detection training operator shown in Figure 1; as shown in Figure 2, this process may include the following steps:
[0071] S201, add 1-10 training images with significant feature differences;
[0072] S202, Create the corresponding number of target type labels according to the actual scenario requirements;
[0073] S203, create a target sample for each type of target;
[0074] S204, through "intelligent appending" or "automatic labeling", marks all targets in the remaining image;
[0075] S205, train the images added to the target training set (based on a deep learning model).
[0076] In the training process of deep learning models, hundreds of sample images are needed for training to improve annotation accuracy and encompass sufficient real-world scenarios, which is time-consuming and laborious. Although ordinary few-shot learning training reduces the number of samples required, repetitive annotation operations are still unavoidable. The embodiments of this application can combine few-shot detection training with automatic annotation or intelligent appending methods to annotate multiple images and multiple target objects, effectively improving the efficiency of target detection annotation in complex scenes.
[0077] Based on the principle shown in Figure 2, this embodiment of the invention provides two specific annotation methods: the first is to annotate the training images by combining manual annotation with automatic annotation and other functions. In some embodiments, it can also be further combined with intelligent appending function for annotation; the second is to annotate the training images by combining manual annotation with intelligent appending function. In some embodiments, it can also be further combined with automatic annotation function for annotation.
[0078] The following sections will provide detailed explanations.
[0079] First, the first method for labeling training images will be explained in detail.
[0080] Referring to Figure 3, which is a flowchart illustrating an embodiment of the first image annotation method provided in this application, the image annotation method provided in this application includes the following steps:
[0081] S301 displays the first training image to be labeled;
[0082] S302, receive user annotations for the first training image, and display the bounding boxes and category labels of different target objects annotated by the user on the first training image; wherein each category label annotates at least one target object;
[0083] S303, in response to the automatic labeling instruction, display on the second training image: the bounding box and category label of the target object in the second training image; wherein, the bounding box and category label of the target object in the second training image are obtained by inputting the second training image into the target detection model trained using the labeled first training image; the labeled first training image includes: user-labeled bounding boxes and category labels.
[0084] In this embodiment, the user can manually annotate the first training image displayed on the electronic device, enabling the electronic device to train the target detection model using the annotated first training image. Then, the user can use an automatic labeling command to input the second training image into the trained target detection model, thereby automatically obtaining the bounding boxes and category labels of the target objects in the second training image. It is evident that by applying this embodiment of the invention, only manual annotation of the first training image is required, while the second training image can be automatically annotated by the electronic device. Therefore, this solution enables more efficient image annotation and reduces annotation time costs.
[0085] In one embodiment, the electronic device receives annotations from a user on a first training image and can display the bounding boxes and category labels of the user-annotated target objects on the first training image; wherein the category labels annotate at least one target object.
[0086] In some embodiments, the first method for annotating training images described above may specifically include:
[0087] Display an interactive interface for labeling training images; the interactive interface includes: an image display area, options for adding training images, options for labeling images, and a "Train" button;
[0088] After the user clicks the option to add a training image, the first training image to be labeled selected by the user is displayed in the image display area.
[0089] After the user clicks the option to annotate the image, the system receives the user's annotations for the first training image and displays the bounding boxes and category labels of the different target objects annotated by the user on the first training image; wherein each category label annotates at least one target object;
[0090] The second training image to be labeled, selected by the user, is displayed in the image display area;
[0091] An automatic labeling control is displayed on the interactive interface. In response to the automatic labeling command generated by the user clicking the automatic labeling control, the bounding boxes and category labels of the target objects in the second training image are displayed on the second training image. The bounding boxes and category labels of the target objects in the second training image are obtained by inputting the second training image into a target detection model trained using a labeled first training image. The labeled first training image includes the bounding boxes and category labels labeled by the user.
[0092] Based on user selection, the bounding boxes and category labels of all target objects after user correction and adjustment are displayed on the second training image;
[0093] After the user clicks the "Train" button, the target detection model is trained using the labeled training images to obtain the trained target detection model.
[0094] In some embodiments, if the training image set contains both a first training image and a second training image, the user can intelligently append more target boxes and category labels to the first training image before automatically labeling the second training image. Then, the second training image is automatically labeled.
[0095] Specifically, referring to Figure 4, which is a flowchart illustrating another embodiment of the first image annotation method provided in this application, the method includes:
[0096] S401, Obtain the training image set specified by the user; wherein, the training image set includes multiple training images;
[0097] In this embodiment, the electronic device can display an interactive interface for annotating training images. Referring to Figure 5(a), Figure 5(a) is a first schematic diagram of the interactive interface used by the image annotation method provided in this application embodiment (the state without imported images). As shown in Figure 5(a), in the "1 / Add Training Image" area, the user can click "Camera Capture" to control the camera connected to the electronic device to acquire training images in real time; or click "Save Image Import" to select an image from the images pre-stored in the electronic device as a training image; or click "External Import" to obtain training images from an external device connected to the electronic device.
[0098] While displaying the interactive interface shown in Figure 5(a), the electronic device can also display a dialog box for setting commonly used parameters for object detection, as shown in Figure 5(b). Figure 5(b) is a second schematic diagram of the interactive interface used by the image annotation method provided in this application embodiment (displaying the status of commonly used parameters for object detection). As shown in Figure 5(b), the dialog box includes a switch control corresponding to angle enable. The user can click the switch control to enable or disable angle enable. When angle enable is enabled, in the training images used for training the object detection model, rectangles with angle values will be used to annotate the target objects; when disabled, rectangles without angles will be used to annotate the target objects. It can be understood that rectangles without angles refer to horizontally and vertically straight rectangles.
[0099] The dialog box may also include a switch control for enabling the optimal model resolution setting. This parameter can be disabled by default. If the user believes the current training image display is unsatisfactory, they can click the switch control to enable this parameter. Once enabled, the dialog box will display the "Optimal Model Resolution" parameter. The user can then set the optimal model resolution value in this dialog box. When the training image is displayed in the subsequent interactive interface, the number of pixels on the longer side of the displayed training image will be adjusted according to the user-set value, while the other side will be scaled proportionally according to the set value. For example, if the value is set to 512, the number of pixels on the longer side of the displayed training image will be 512.
[0100] When the target detection model to be trained is a few-shot target detection model, if the first method of annotating training images is used, the user can select 2-10 training images as the training set; of course, more than 10 training images are also acceptable. Users should try to select images with significant feature differences to increase sample diversity and thus improve the training effect of the target detection model. Each training image can contain only one type of target object or multiple types. If the second method of annotating training images is used, the user can select 1-10 training images as the training set; the specific process will be explained in detail below.
[0101] S402, Receive a training image selected by the user from the training image set as the first training image to be labeled; wherein, other images in the training image set other than the first training image to be labeled are used as the second training images to be labeled.
[0102] Once the user has determined the training image set, they can select an image that includes all categories of target objects as the first training image. By obtaining the user-determined training image set and receiving the training image selected by the user from the training image set as the first training image to be labeled, a foundation can be provided for subsequent labeling of the first and second training images.
[0103] In another implementation, the electronic device can receive a training image selected by the user from the training image set as the first training image to be labeled; and, under the condition that preset automatic labeling conditions are met, determine at least one image from the other images in the training image set besides the first training image as the second training image to be labeled. That is, the first training image can be imported first, and then the second training image can be imported when the preset automatic labeling conditions are met. The preset automatic labeling conditions can be user-triggered automatic labeling commands, etc., and are not specifically limited here.
[0104] S403 displays the first training image to be labeled;
[0105] In this step, the user can select an image as the first training image after obtaining the user-defined training image set. After the user selects the first training image, the first training image can be displayed in the user interface of the electronic device. For other training images not currently displayed, the electronic device can display their thumbnails in the thumbnail display area. Referring to Figure 6, Figure 6 is a third schematic diagram of the interactive interface shown in Figure 5(a) (the state of importing images). As shown in Figure 6, the thumbnail corresponding to the currently displayed training image can display an outline effect to indicate to the user which training image is currently displayed.
[0106] S404, in response to the label creation instruction, receives and saves the different category labels corresponding to different target objects created by the user for the first training image;
[0107] Before responding to a tag creation command, the electronic device can display a tag creation control; thus, the tag creation command can be generated by the user clicking the tag creation control. Referring again to Figure 6 as an example, the tag creation control can be "New" in the "Tag List" area. Afterwards, the electronic device can display a tag creation dialog box, as shown in Figure 7. Figure 7 is a fourth schematic diagram of the interactive interface shown in Figure 5(a) (the state when creating a tag). As shown in Figure 7, the user can enter the tag name and set the tag color in this dialog box, so that different category tags can be displayed in different colors for easy user identification.
[0108] For example, if the first training image contains two different target objects: a large pill and a small pill, the user can create two category labels, naming them "large pill" and "small pill" respectively. After the user creates the category labels, the label control corresponding to the created category label can be displayed below "Label Category", as shown in Figure 8. Figure 8 is the fifth schematic diagram of the interactive interface shown in Figure 5(a) (the state after the label is created). As shown in Figure 8, the label control corresponding to each category label can also include "edit" and "delete" buttons, used to modify the name of the category label and delete the category label, respectively.
[0109] This embodiment receives instructions by displaying controls, which makes it easier for users to trigger the corresponding functions and improves the user experience.
[0110] S405, receive user annotations for the first training image, and display the bounding boxes and category labels of different target objects annotated by the user on the first training image; wherein, each category label annotates at least one target object;
[0111] Taking Figure 8 as an example, users can click "Rectangle Selection" in the "2 / Annotated Image" area to annotate the target objects in the first training image. For example, users can draw a rectangle as the target box at the location of the target object and surround the target object inside the rectangle. The electronic device will then display the target box drawn by the user and display the label category of the target object near the target box.
[0112] Before receiving user annotations for the first training image, the electronic device can first display label controls corresponding to different target objects created by the user. In this case, the bounding box and category label of each target object annotated by the user on the first training image are obtained when the user selects the label control corresponding to that target object.
[0113] When multiple target objects exist in the first training image, the user can first select the label control corresponding to one type of target object. The selected label control can also be displayed in the form of an outline. Then, the user can label the target object in the training image. Afterwards, the user can also select the label control corresponding to other target objects to label other categories of target objects, thereby achieving the labeling of target objects of different categories. See Figure 9, which is the sixth schematic diagram of the interactive interface shown in Figure 5(a) (the state after the user has labeled the first training image). As shown in Figure 9, the user can label a "large pill" and a "small pill" in the image respectively. At this time, the target bounding box and category label labeled by the user can be used as target examples. For each category of target object, the user only needs to label one. Of course, the user can also label multiple target objects for each category to effectively enrich the target examples and make subsequent additional labels more accurate.
[0114] S406, in response to the label append instruction, displays on the first training image: the bounding box and category label of the target object in the first training image;
[0115] The bounding boxes and category labels of the target objects in the first training image are obtained by inputting the first training image into a target detection model trained using the user-annotated first training image.
[0116] Before responding to a tag append command, the electronic device can first display a smart append control; thus, the tag append command can be generated by the user clicking the smart append control. After the user clicks the smart append control, there is a certain time delay in responding to the tag append command. During this period, the electronic device can also provide some loading and waiting prompts, such as displaying a dialog box containing text to prompt the user.
[0117] In one implementation, responding to a label appending instruction includes: responding to a label appending instruction when the smart appending control is unlocked; wherein the unlocking condition of the smart appending control is: the presence of a user-annotated bounding box and category label on the first training image.
[0118] In other words, when the training dataset does not yet contain user-annotated bounding boxes and category labels, the smart append control can be locked. For example, electronic devices can temporarily hide the smart append control, or display it in a lighter color, until it is unlocked and then displayed normally. By locking and unlocking the control, users can be prevented from accidentally activating it when its corresponding function is not working.
[0119] If unlabeled target objects still exist in the first training image labeled by the user, the user can issue a label appending command. After receiving the label appending command, the electronic device can first train the target detection model based on the target bounding boxes and category labels included with the user's annotations, and then input the first training image into the target detection model to obtain the target bounding boxes and category labels of the target objects detected by the target detection model in the first training image and display them.
[0120] Object detection models can be trained using a few-shot testing method. For example, the training process for an object detection model can involve the following steps:
[0121] First, the unlabeled first training image is input into the target detection model to be trained, and the bounding boxes and class labels of the target objects detected by the model in the image are obtained as the model prediction results. The model loss is calculated based on the model prediction results and the ground truth of the first training image, and the model parameters are tuned using the calculated model loss until the model converges, and the trained target detection model is obtained. The ground truth of the first training image can be the labeled bounding boxes and class labels in the first training image.
[0122] S407 displays the second training image to be labeled;
[0123] After annotating the first training image, the user can switch to the second training image, which will then be displayed on the electronic device. For example, as shown in Figure 6, the user can click on the thumbnail of a currently undisplayed training image in the thumbnail display area to switch training images.
[0124] S408, in response to the automatic labeling instruction, displays on the second training image: the bounding box and category label of the target object in the second training image;
[0125] The bounding boxes and category labels of the target objects in the second training image are obtained by inputting the second training image into a target detection model trained using the labeled first training image. The labeled first training image includes user-annotated bounding boxes and category labels. Of course, if the electronic device responds to a label appending command for the first training image, the labeled first training image also includes the appended bounding boxes and category labels.
[0126] Similarly, before responding to an automatic marking instruction, the electronic device can first display an automatic marking control, so that the automatic marking instruction can be generated by the user by clicking the automatic marking control.
[0127] The steps for responding to an automatic labeling command may include: responding to an automatic labeling command when the automatic labeling control is unlocked; wherein the unlocking condition of the automatic labeling control is: the first training image has been labeled and the second training image to be labeled is displayed.
[0128] In one implementation, referring to Figure 10, the seventh schematic diagram of the interactive interface shown in Figure 5(a) (displaying the state of the automatic labeling button), as shown in Figure 10, can generate a non-modal pop-up (Toast) prompt box while displaying the second training image. Simultaneously, the automatic labeling control (i.e., "Train and Automatic Label" in the figure) is displayed in this prompt box, along with the text: "The image can be automatically labeled through automatic inference." Of course, the specific presentation of the automatic labeling control can also be defined according to the business scenario; this is merely an example.
[0129] After receiving the automatic labeling instruction, the electronic device can first train the object detection model using the already labeled first training image. Then, it can input the second training image into the trained object detection model to obtain the bounding boxes and category labels of the target objects in the second training image and display them. Since this process takes some time, the electronic device can also provide a loading and waiting prompt during this period.
[0130] In one specific implementation, step S408 may include:
[0131] In response to the automatic labeling command, the bounding boxes and category labels of the target objects annotated after the second training image is input into the target detection model are displayed on the second training image; and, based on the user selection, the bounding boxes and category labels of all target objects after user correction and adjustment are displayed on the second training image.
[0132] Specifically, the correction and adjustment process can include the following situations:
[0133] Scenario 1: When a modification instruction is received for any target box, the position, size, and / or corresponding category label of the target box are modified based on the modification instruction;
[0134] Scenario 2: When a deletion instruction is received for any target box, the target box and its corresponding category label are removed from the training image;
[0135] Scenario 3: When a user adds new annotations to a training image, the user's new bounding box and category label are displayed on the training image.
[0136] In this embodiment, the electronic device allows users to modify and adjust the bounding box and category label of the target object, which can further ensure the accuracy of the annotation and thus improve the training effect of the target detection model.
[0137] When the number of second training images to be labeled is greater than one, after executing the automatic labeling instruction on the first second training image, the method may further include:
[0138] S409: Display the next second training image to be labeled in sequence, and execute automatic labeling instructions for the second training image until all second training images are labeled.
[0139] In other words, steps S407-S408 can be executed multiple times until all second training images are labeled. This allows for rapid labeling of individual training images even when there are multiple images. Therefore, this solution provides an automatic labeling function applicable to multi-image scenarios, quickly identifying existing targets of the same type in different images. It effectively saves labeling time in complex scenes with a large number of duplicate targets, thus improving labeling efficiency.
[0140] Furthermore, this embodiment combines three methods—pre-training with few samples for deep learning detection, intelligent appending, and automatic annotation—to reduce the number of samples requiring manual annotation from hundreds or thousands to just a few or tens. This significantly reduces the time and effort spent on manual annotation during target detection training, and also greatly reduces the time and manpower costs of deploying practical applications.
[0141] In this embodiment, the user can create tags in the electronic device and manually annotate the first training image displayed by the electronic device. This allows the electronic device to train an object detection model using the annotated first training image. The electronic device can then automatically annotate the target objects in the first training image that the user has not yet labeled. After displaying the second training image, the user can use an automatic labeling command to input the second training image into the trained object detection model, thereby automatically obtaining the bounding boxes and category labels for the target objects in the second training image. It is evident that this solution only requires manual annotation of a portion of the target objects in the first training image; the remaining target objects in the first training image, as well as the target objects in the second training image, can be automatically labeled by the electronic device. Therefore, this solution enables more efficient image annotation and reduces the time cost of annotation.
[0142] In one embodiment of this application, referring to FIG11, FIG11 is a schematic diagram of the intelligent appending process in the embodiment shown in FIG4. As shown in FIG11, the process may include the following steps:
[0143] S1101, in response to the label appending instruction, display the bounding box and category label of the target object to be appended in this time on the first training image; wherein, the bounding box and category label of the target object to be appended in this time are obtained by inputting the first training image into the target detection model trained using the first training image labeled by the user, and the similarity between the target object to be appended in this time and the target object labeled by the user is greater than a preset threshold.
[0144] The similarity between two target objects can refer to the image similarity between the local images bounded by the bounding boxes of the two target objects. In other words, after the electronic device responds to the label appending instruction, it may not append bounding boxes and category labels to all target objects detected by the object detection model, but only to target objects whose similarity to the user-labeled target objects is greater than a preset threshold. Specifically, the electronic device can calculate the similarity between each target object in each category detected by the object detection model and the currently labeled target objects of that category. If there are multiple currently labeled target objects of that category, the average similarity can be taken. If the similarity is greater than the preset threshold, the bounding box and category label of that target object are appended; otherwise, no additional labeling is added.
[0145] The preset threshold can be set according to the actual situation and experience. In one implementation, the preset threshold can be set higher, such as 98%, so that only 1-3 target objects are automatically added each time. This number makes it easy for users to confirm and adjust to gradually release a small number of inference targets, which helps to continuously improve the accuracy of inference.
[0146] For example, see Figure 12(a). Figure 12(a) is a schematic diagram of the labeling method for images provided in the embodiments of this application. After the first labeling is performed on the first training image in Figure 9, the resulting first training image can be as shown in Figure 12(a).
[0147] After each additional annotation, the electronic device can also modify and adjust the bounding box and category label of the target object of the additional annotation based on the user's selection.
[0148] After the label appending instruction is executed, step S1102 is executed: determine whether the bounding boxes and category labels of all target objects are displayed on the first training image; if not, execute S1103; if yes, execute S1104.
[0149] If the bounding boxes and category labels of all target objects are not displayed on the first training image, the method may further include:
[0150] S1103, in response to the next label appending instruction, display the bounding box and category label of the target object to be appended in this time on the first training image; wherein, the bounding box and category label of the target object to be appended in this time are obtained by inputting the first training image into the updated target detection model, and the similarity between the target object to be appended in this time and the already labeled target object is greater than a preset threshold.
[0151] The updated object detection model is trained on the first training image, which includes user-annotated bounding boxes and category labels, as well as previously appended bounding boxes and category labels.
[0152] In other words, users can issue label appending commands multiple times until all target objects on the first training image are labeled, or until the object detection model can no longer detect new target objects. Furthermore, before each label appending command, the electronic device can retrain the object detection model using the user-annotated bounding boxes and category labels, as well as the first training image with previously appended bounding boxes and category labels, to obtain an updated object detection model.
[0153] For example, if there are 10 target objects in the first training image, the user can manually label one target object in the first training image and then issue a label appending command. In response to this label appending command, the electronic device adds labels to 3 target objects, and the user can then issue another label appending command. In response to this label appending command, the electronic device adds labels to 2 more target objects, and so on, until the electronic device has appended labels to all target objects, at which point the user can stop issuing label appending commands.
[0154] For example, see Figure 12(b), which is a schematic diagram of the target object being fully annotated in the image annotation method provided in the embodiments of this application.
[0155] S1104 completes the intelligent addition process.
[0156] In this embodiment, the electronic device can gradually release the results of the additional annotations, leaving enough room for the user to gradually correct the target detection model. This can significantly improve the accuracy of the additional annotation process and reduce detection errors caused by external factors such as the environment.
[0157] For ease of understanding, please refer to Figure 13, which is a schematic diagram of the intelligent addition interaction principle of the embodiment shown in Figure 4. As shown in Figure 13, the process includes the following steps:
[0158] S1301, Smart Addition Function is disabled by default; that is, the Smart Addition Control is not unlocked in the initial interactive interface displayed on the electronic device.
[0159] S1302, at least one target has been marked in the current image. At this time, the conditions for unlocking the smart append function have been met, so S1303 is executed.
[0160] S1303, the intelligent add-on function is unblocked from the disabled state;
[0161] S1304, click "Smart Add"; that is, the user clicks "Smart Add".
[0162] S1305 invokes the algorithm to mark the most similar targets (i.e., target objects) located on the image based on the existing annotations, with a number between 1 and 3.
[0163] This step is the process described above: training the target detection model using the labeled first training image, and then inputting the first training image into the target detection model.
[0164] S1306, Determine whether the appended content is correct, that is, the user determines whether the target object intelligently appended is correct; if yes, then execute S1307; if no, then execute S1308.
[0165] S1307, determine whether all targets in the current image have been appended; that is, the user determines whether all targets in the current image have been appended; if yes, execute S1309; if no, return to execute 1304;
[0166] S1308, Correct and adjust the marked content; that is, the user can correct and adjust the target box and category label of the marked target object.
[0167] S1309, switch to the next image.
[0168] In one embodiment of this application, referring to FIG14, FIG14 is a schematic diagram of the automatic annotation process in the embodiment shown in FIG4. As shown in FIG14, the process may include the following steps:
[0169] S1401 displays the first second training image to be labeled;
[0170] S1402, in response to the automatic labeling instruction, displays the bounding boxes and category labels of the target objects annotated after the second training image is input into the target detection model on the second training image; and, based on the user selection, displays the bounding boxes and category labels of all target objects after user correction and adjustment on the second training image.
[0171] After each automatic labeling instruction is executed on a second training image, the image labeling method provided in this application embodiment may further include:
[0172] S1403, Based on user selection, the second training image, which has been corrected and adjusted by the user, is added to the target training set to update the target training set;
[0173] Specifically, after the user has labeled a second training image, the user can decide whether to add the second training image to the target training set. If the user decides to add the second training image to the target training set, the electronic device can first determine whether the second training image has been manually corrected and adjusted by the user. If the second training image has not been corrected and adjusted, the electronic device can refuse to add the second training image to the target training set and display a text prompt to the user. For example, see Figure 15, which is the eighth schematic diagram of the interactive interface shown in Figure 5(a) (the state of prompting the refusal to add the image to the target training set). As shown in Figure 15, the text "The automatically labeled image will not participate in training. Please add it to the training samples after manual adjustment" can be displayed below the second training image.
[0174] If the second training image is corrected and adjusted, it can be added to the target training set.
[0175] S1404 shows the next second training image to be labeled;
[0176] After each display of the next second training image to be labeled, in response to the automatic labeling instruction, the labeling method for the image may further include:
[0177] S1405, Based on user selection, train the object detection model with the updated object training set to update the object detection model;
[0178] In other words, after each second training image is displayed, the user can decide whether to update the object detection model. If an update is needed, the electronic device can use all images in the current target training set to update the object detection model; if no update is needed, the object detection model that was automatically labeled on the previous second training image can continue to be used.
[0179] For the second and subsequent second training images, in response to the automatic labeling command, the above-mentioned bounding boxes and category labels of the target objects in the second training images are displayed on the second training images, including:
[0180] S1406, based on user selection, display the bounding boxes and category labels of the target objects annotated after the target detection model has been updated or not, on the second training image; and, based on user selection, display the bounding boxes and category labels of all target objects after user correction and adjustment on the second training image.
[0181] Referring to Figure 16, which is the ninth schematic diagram of the interactive interface shown in Figure 5(a), the electronic device can display a Toast notification box after each display of the next second training image, with two controls displayed in the notification box: "Auto-label" and "Retrain and Auto-label". Of course, this process is not limited to this display method; this is just an example. If the user clicks "Auto-label", it means that the object detection model does not need to be updated, and the second training image is input from the unupdated object detection model. If the user clicks "Retrain and Auto-label", it means that the object detection model needs to be updated, and the second training image is input from the updated object detection model.
[0182] For ease of understanding, please refer to Figure 17, which is a schematic diagram of the automatic marking interaction principle of the embodiment shown in Figure 4. As shown in Figure 17, the process may include the following steps:
[0183] S1701, automatic marking function is disabled by default;
[0184] That is, the automatic labeling function is not unlocked in the initial interactive interface displayed on the electronic device;
[0185] S1702, confirming that content has already been labeled on an image;
[0186] If an image has already been labeled, switch to an image without labels to unlock the automatic labeling function and execute S1703.
[0187] S1703, the automatic marking function is deactivated and related instructions are displayed on the interactive interface; see Figure 10 for related instructions.
[0188] S1704, click "Train and Auto Label";
[0189] That is, the user clicks "Train and Automatic Labeling";
[0190] S1705, Mark all the inferred targets on the image;
[0191] That is, the electronic device trains the model based on the existing labeled content, and then marks all the inferred target objects on the image.
[0192] The first execution of this step involves training the object detection model using the labeled first training image, and then inputting the second training image into the object detection model. On subsequent executions, this step involves displaying the bounding boxes and category labels of the target objects labeled with the second training image (either the one input into the unupdated or updated object detection model) based on user selection.
[0193] S1706, determine whether to adjust the automatic annotation; that is, the user can determine whether to adjust the automatic annotation; if yes, then execute S1707; if no, then execute S1708.
[0194] S1707, Adjust or supplement the marked content or targets that are not automatically marked; that is, the user adjusts the target box and category label of the marked target objects, or supplements the target objects that are not automatically marked.
[0195] S1708, Confirm whether to include the current image in the target training set; that is, the user confirms whether to include the current image in the target training set; if yes, proceed to S1707, at which point the interactive interface will display a prompt message if the image has not been manually adjusted, to remind the user that the image to be included in the target training set needs to be manually adjusted. If no, proceed to S1709;
[0196] S1709, Switch to the next unlabeled image;
[0197] S1710, Confirm whether to use the current model for automatic labeling; that is, when switching to the next image, the interactive interface will display a prompt message to ask the user to confirm whether to use the current model for automatic labeling, as shown in Figure 16. If yes, proceed to S1711; if no, proceed to S1712;
[0198] S1711, continue automatic labeling with the current model; at this time, the user can execute S1713: click "Automatic Labeling", and then the interaction logic will return to S1705 again;
[0199] S1712, retrain the new model based on the current target training set and then perform automatic labeling; that is, update the target detection model and use the updated target detection model for automatic labeling; at this time, the user can execute S1714: click "Retrain and Automatically Label", and then the interaction logic will return to S1705 again. By repeating this cycle, all images can be labeled.
[0200] In this embodiment, the user can filter the training images to determine whether to add them to the target training set. Furthermore, the user can use the images in the target training set to update the target detection model, thereby continuously improving the performance of the target detection model and making the automatic annotation function more and more accurate.
[0201] The second method for labeling training images will be explained in detail below.
[0202] This application also provides an image annotation method for use in electronic devices. Referring to Figure 18, Figure 18 is a flowchart illustrating an embodiment of a second image annotation method provided in this application. As shown in Figure 18, the method includes:
[0203] S1801 displays a training image to be labeled;
[0204] S1802, Receive user annotations for the training image, and display the bounding boxes and category labels of different target objects annotated by the user on the training image;
[0205] Each category label identifies at least one target object;
[0206] S1803, in response to the label append instruction, display on the training image: the bounding box and category label of the target object in the training image;
[0207] The bounding boxes and category labels of the target objects in the training image are obtained by inputting the training image into a target detection model trained using user-annotated training images.
[0208] The steps above are similar to S403-S406, and the similarities will not be repeated here. The difference between this embodiment and the embodiment shown in Figure 4 is that in this embodiment, image annotation can be completed through intelligent appending function without using automatic annotation function.
[0209] In this embodiment, only some target objects in the training image need to be labeled manually, while other target objects can be labeled automatically by electronic devices. Therefore, this solution can label images more efficiently and reduce the time cost of labeling.
[0210] In one embodiment, the electronic device receives annotations from a user on a training image and can display bounding boxes and category labels of the user-annotated target objects on the training image; wherein the category labels annotate at least one target object.
[0211] In some embodiments, the second method for annotating training images described above may specifically include:
[0212] Display an interactive interface for labeling training images; the interactive interface includes: an image display area, options for adding training images, options for labeling images, and a "Train" button; wherein, the options for labeling images include: "rectangular selection" and "intelligent appending";
[0213] After the user clicks the option to add a training image, a training image selected by the user to be labeled is displayed in the image display area.
[0214] After the user clicks "rectangular selection", the system receives the user's annotations for the training image and displays the target bounding boxes and category labels of different target objects annotated by the user on the training image; wherein each category label annotates at least one target object;
[0215] In response to the user's command to add tags by clicking "Smart Add", the bounding boxes and category labels of the target objects in the training image are displayed on the training image; wherein, the bounding boxes and category labels of the target objects in the training image are obtained by inputting the training image into a target detection model trained using user-annotated training images;
[0216] Based on user selection, the target bounding boxes and category labels of the newly added objects were corrected and adjusted.
[0217] After the user clicks the "Train" button, the target detection model is trained using the labeled training images to obtain the trained target detection model.
[0218] Referring to Figure 19, which is a flowchart illustrating another embodiment of the second image annotation method provided in this application, the method includes:
[0219] S1901 displays a training image to be labeled;
[0220] S1902, Receive user annotations for the training image, and display the bounding boxes and category labels of different target objects annotated by the user on the training image; wherein, each category label annotates at least one target object;
[0221] S1903, in response to the label append instruction, display on the training image: the bounding box and category label of the target object in the training image;
[0222] The bounding boxes and category labels of the target objects in the training image are obtained by inputting the training image into a target detection model trained using user-annotated training images.
[0223] The above steps are similar to those in S403-S406, and the similarities will not be repeated here.
[0224] After step S1903, steps S1102-S1103 can be executed until the bounding boxes and category labels of all target objects are displayed on the current training image, and then step S1904 is executed.
[0225] S1904, determine whether there are still training images to be labeled; if not, proceed to S1906; if yes, proceed to S1905.
[0226] S1905, display the next training image to be labeled; then you can return to execute step S1902.
[0227] S1906, completes the intelligent addition process.
[0228] In other words, when there are multiple training images to be labeled, after executing the label appending instruction for the first training image, the next training image to be labeled can be displayed sequentially. After displaying the next training image to be labeled, it can be manually labeled, and then the label appending instruction can be executed for that training image again. This cycle continues until all training images are labeled.
[0229] In this embodiment, only some target objects in the training image need to be labeled manually, while the other target objects in the training image can be labeled automatically by electronic devices. Therefore, this solution can label images more efficiently and reduce the time cost of labeling.
[0230] In one embodiment, in the second image annotation method provided in this application, the user can also annotate the training image through automatic annotation. See Figure 20 for details. Figure 20 is a flowchart illustrating another embodiment of the second image annotation method provided in this application. As shown in Figure 20, the method includes:
[0231] S2001 displays a training image to be labeled;
[0232] S2002, Receive user annotations for the training image, and display the bounding boxes and category labels of different target objects annotated by the user on the training image; wherein, each category label annotates at least one target object;
[0233] S2003, in response to the label appending instruction, displays on the training image: the bounding box and category label of the target object in the training image;
[0234] The bounding boxes and category labels of the target objects in the training image are obtained by inputting the training image into a target detection model trained using user-annotated training images.
[0235] The above steps are similar to those in S403-S406, and the similarities will not be repeated here.
[0236] After step S2003, steps S1102-S1103 can also be executed until the bounding boxes and category labels of all target objects are displayed on the current training image, and then step S2004 is executed.
[0237] S2004, determine whether there are still training images to be labeled; if not, proceed to S2007; if yes, proceed to S2005.
[0238] S2005, Display the next training image to be labeled;
[0239] S2006, in response to the automatic labeling instruction, display the bounding boxes and class labels of the target objects in the next training image; wherein, the bounding boxes and class labels of the target objects in the training image are obtained by inputting the training image into the target detection model trained using the labeled training images; then, the process can return to step S2004 until all training images are labeled.
[0240] S2007 completes the automatic annotation process.
[0241] Based on the above image annotation method, this application embodiment also provides a model training method. Referring to Figure 21, Figure 21 is a flowchart of another embodiment of the model training method provided by this application embodiment. As shown in Figure 21, the method includes the following steps:
[0242] S2101, Obtain the training image specified by the user;
[0243] S2102, the obtained training images are labeled using any of the image annotation methods described above;
[0244] S2103, in response to the model training instruction, trains the object detection model using the labeled training images to obtain the trained object detection model.
[0245] Model training commands can also be issued by the user by clicking the corresponding control in the interactive interface, for example, the control could be the "Train" button in Figure 6.
[0246] This application does not limit the specific training method. For example, training can be performed as follows:
[0247] First, input the unlabeled training image into the target detection model to be trained to obtain the target bounding box and category label of the target object detected by the model in the image, which are used as the model prediction results. Calculate the model loss based on the model prediction results and the labeled results of the training image, and use the calculated model loss to tune the model parameters of the target detection model until the model converges, thus obtaining the trained target detection model.
[0248] In this embodiment, only a portion of the target objects in the training images need to be manually labeled. The remaining target objects in the training images, as well as the target objects in other training images, can be automatically labeled by electronic devices. Therefore, this solution can label images more efficiently, reduce the time cost of labeling, and thus improve the efficiency of target detection model training.
[0249] This application also provides an image annotation device for use in electronic devices. Referring to FIG22, FIG22 is a schematic structural diagram of an embodiment of the image annotation device provided in this application. As shown in FIG22, the device includes:
[0250] Display module 2201 is used to display the first training image to be labeled;
[0251] The annotation receiving module 2202 is used to receive annotations from the user for the first training image and display the bounding boxes and category labels of different target objects annotated by the user on the first training image; wherein each category label annotates at least one target object;
[0252] The automatic labeling response module 2203 is used to display, in response to an automatic labeling command, the bounding boxes and category labels of the target objects in the second training image on the second training image; wherein the bounding boxes and category labels of the target objects in the second training image are obtained by inputting the second training image into a target detection model trained using a labeled first training image; the labeled first training image includes user-labeled bounding boxes and category labels.
[0253] In one embodiment, the apparatus further includes a label appending response module, configured to, in response to a label appending instruction, display on the first training image: the target bounding boxes and category labels of the target objects in the first training image, after the label receiving module 2202 receives the user's annotations for the first training image and displays the target bounding boxes and category labels of the different target objects annotated by the user on the first training image; wherein, the target bounding boxes and category labels of the target objects in the first training image are obtained by inputting the first training image into a target detection model trained using the user-annotated first training image; the annotated first training image further includes: the target bounding boxes and category labels annotated after appending.
[0254] In one embodiment, the device further includes:
[0255] The label creation response module is used to respond to the label creation instruction before the label receiving module 2202 receives the user's label for the first training image, and to receive and save the different category labels corresponding to different target objects created by the user for the first training image.
[0256] In one embodiment, the device further includes an image switching module, which, when the number of second training images to be labeled is greater than 1, sequentially displays the next second training image to be labeled after the automatic labeling response module 2203 has executed the automatic labeling instruction on the first second training image, and executes the automatic labeling instruction on the second training image until all second training images are labeled.
[0257] In one embodiment, the apparatus further includes: a training image set acquisition module, configured to acquire a user-defined training image set before the display module 2201 displays the first training image to be labeled; wherein the training image set includes multiple training images;
[0258] The training image receiving module is used to receive a training image selected by the user from the training image set as the first training image to be labeled; wherein, apart from the first training image to be labeled, the other images in the training image set are used as the second training images to be labeled.
[0259] In one embodiment, the automatic marking response module 2203 is specifically used for:
[0260] For the first second training image, in response to the automatic labeling instruction, the bounding boxes and category labels of the target objects annotated after the second training image is input into the target detection model are displayed on the second training image; and, based on the user selection, the bounding boxes and category labels of all target objects after user correction and adjustment are displayed on the second training image.
[0261] In one embodiment, the apparatus further includes a target training set update module, which, after each automatic labeling instruction is executed on a second training image, adds the second training image, which has been corrected and adjusted by the user, to the target training set based on the user's selection, so as to update the target training set;
[0262] The model update module is used to train the target detection model with the updated target training set based on the user selection before the automatic labeling response module 2203 responds to the automatic labeling instruction after the image switching module displays the next second training image to be labeled each time.
[0263] Automatic marking response module 2203 is specifically used for:
[0264] Based on user selection, the bounding boxes and category labels of the target objects annotated after the target detection model has been updated or not, are displayed on the second training image; and based on user selection, the bounding boxes and category labels of all target objects after user correction and adjustment are displayed on the second training image.
[0265] In one embodiment, the tag appending response module is specifically used for:
[0266] In response to the label appending instruction, the bounding box and category label of the target object to be appended are displayed on the first training image; wherein, the bounding box and category label of the target object to be appended are obtained by inputting the first training image into the target detection model trained using the first training image labeled by the user, and the similarity between the target object to be appended and the target object labeled by the user is greater than a preset threshold.
[0267] In one embodiment, the label appending response module is further configured to, in response to the next label appending instruction, display the bounding boxes and category labels of the newly appended target objects on the first training image when the first training image does not display the bounding boxes and category labels of all target objects after the label appending instruction is executed; wherein the bounding boxes and category labels of the newly appended target objects are obtained by inputting the first training image into the updated target detection model, and the similarity between the newly appended target objects and the already labeled target objects is greater than a preset threshold; the updated target detection model is obtained by training the target detection model based on the first training image containing the user-annotated bounding boxes and category labels, as well as the previously appended bounding boxes and category labels.
[0268] In one embodiment, the device further includes:
[0269] The correction and adjustment module is used to correct and adjust the target box and category label of the target object to be added this time based on the user's selection after the label appending response module displays the target box and category label of the target object to be added this time.
[0270] In one embodiment, the device further includes:
[0271] The first control display module is used to display a tag creation control before the tag creation response module responds to the tag creation instruction; wherein the tag creation instruction is generated by the user by clicking the tag creation control;
[0272] The second control display module is used to display the label controls corresponding to different target objects created by the user before the label receiving module 2202 receives the user's annotations for the first training image; wherein, the target box and category label of each target object annotated by the user on the first training image are obtained by annotating when the user selects the label control corresponding to the target object;
[0273] The third control display module is used to display the automatic marking control before the automatic marking response module 2203 responds to the automatic marking instruction; wherein the automatic marking instruction is generated by the user by clicking the automatic marking control;
[0274] The fourth control display module is used to display a smart append control before the tag append response module responds to the tag append instruction; wherein the tag append instruction is generated by the user by clicking the smart append control.
[0275] In one embodiment, the automatic labeling response module 2203 is specifically used to: respond to an automatic labeling command when the automatic labeling control is unlocked; wherein, the unlocking condition of the automatic labeling control is: the first training image has been labeled, and a second training image to be labeled is displayed;
[0276] The label appending response module is specifically used to: respond to a label appending command when the smart appending control is unlocked; wherein the unlocking condition of the smart appending control is: there are user-annotated target boxes and category labels on the first training image.
[0277] In one embodiment, the device further includes:
[0278] The modification response module is used to modify the position, size, and / or corresponding category label of any target box based on the modification instruction received when a modification instruction for any target box is received.
[0279] The deletion response module is used to remove the target box and its corresponding category label from the training image when a deletion instruction is received for any target box;
[0280] The annotation module is used to display the user-added bounding boxes and category labels on the training image when the module receives new annotations from the user.
[0281] In some embodiments, the image annotation device described above may specifically include:
[0282] The display module is used to display an interactive interface for labeling training images; the interactive interface includes: an image display area, options for adding training images, options for labeling images, and a "Train" button; after the user clicks the option to add training images, the first training image to be labeled selected by the user is displayed in the image display area.
[0283] The annotation receiving module is used to receive the annotations of the user for the first training image after the user clicks the option to annotate the image, and to display the target boxes and category labels of different target objects annotated by the user on the first training image; wherein each category label annotates at least one target object;
[0284] An automatic labeling response module is used to display the second training image to be labeled selected by the user in the image display area; in response to the automatic labeling instruction generated by the user by clicking the automatic labeling control, it displays on the second training image: the bounding boxes and category labels of the target objects in the second training image; based on the user selection, it displays on the second training image the bounding boxes and category labels of all target objects after user correction and adjustment; wherein, the bounding boxes and category labels of the target objects in the second training image are obtained by inputting the second training image into a target detection model trained using the labeled first training image; the labeled first training image includes: the bounding boxes and category labels labeled by the user;
[0285] The model training instruction response module is used to train the object detection model using the labeled training images after the user clicks the "Train" button, so as to obtain the trained object detection model.
[0286] This application also provides an image annotation device for use in electronic devices. Referring to Figure 23, Figure 23 is a schematic diagram of another embodiment of the image annotation device provided in this application; the device includes:
[0287] The training image display module 2301 is used to display a training image to be labeled;
[0288] User annotation receiving module 2302 is used to receive user annotations for the training image and display the bounding boxes and category labels of different target objects annotated by the user on the training image; wherein each category label annotates at least one target object;
[0289] The label appending instruction response module 2303 is used to display, in response to a label appending instruction, the bounding box and category label of the target object in the training image on the training image; wherein the bounding box and category label of the target object in the training image are obtained by inputting the training image into a target detection model trained using user-annotated training images.
[0290] In one embodiment, the number of training images to be labeled is multiple; the apparatus further includes:
[0291] The image appending module is used to display the next training image to be labeled sequentially after executing the label appending instruction for the first training image, and execute the label appending instruction for that training image until all training images are labeled.
[0292] In one embodiment, the number of training images to be labeled is multiple; the apparatus further includes:
[0293] The automatic labeling module is used to sequentially display the next training image to be labeled after executing the label appending instruction for the first training image; in response to the automatic labeling instruction, it displays the bounding box and category label of the target object in the next training image; wherein the bounding box and category label of the target object in the training image are obtained by inputting the training image into the target detection model trained using the labeled training images; and returns to execute the step of sequentially displaying the next training image to be labeled until all training images are labeled.
[0294] In some embodiments, the image annotation device described above may specifically include:
[0295] The training image display module is used to display an interactive interface for labeling training images; the interactive interface includes: an image display area, options for adding training images, options for labeling images, and a "Train" button; wherein, the options for labeling images include: "rectangular selection" and "intelligent appending";
[0296] After the user clicks the option to add a training image, a training image selected by the user to be labeled is displayed in the image display area;
[0297] The user annotation receiving module is used to receive the user's annotations for the training image after the user clicks "rectangular selection", and to display the target boxes and category labels of different target objects annotated by the user on the training image; wherein each category label annotates at least one target object;
[0298] The label appending instruction response module is used to respond to the label appending instruction generated by the user by clicking "Smart Append" and display the target bounding box and category label of the target object in the training image on the training image; and to correct and adjust the target bounding box and category label of the target object to be appended based on the user's selection; wherein, the target bounding box and category label of the target object in the training image are obtained by inputting the training image into the target detection model trained using the training image labeled by the user;
[0299] The model training instruction response module is used to train the object detection model using the labeled training images after the user clicks the "Train" button, so as to obtain the trained object detection model.
[0300] This application embodiment also provides a model training device applied to an electronic device. Referring to Figure 24, Figure 24 is a schematic diagram of the structure of the model training device provided in this application embodiment. As shown in Figure 24, the device includes:
[0301] Training image acquisition module 2401 is used to acquire training images determined by the user;
[0302] The training image annotation module 2402 is used to annotate the obtained training images using any of the image annotation methods described above.
[0303] The model training instruction response module 2403 is used to respond to the model training instruction and train the object detection model using the labeled training images to obtain the trained object detection model.
[0304] This application also provides an electronic device, as shown in FIG25. FIG25 is a schematic diagram of the structure of the electronic device provided in this application embodiment, as shown in FIG25, including:
[0305] Memory 2501 is used to store computer programs;
[0306] The processor 2502, when executing the program stored in the memory 2501, implements the steps of the above-mentioned image annotation method or model training method.
[0307] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 2502, the communication interface, and the memory 2501 communicating with each other via the communication bus.
[0308] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0309] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0310] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0311] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0312] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described image annotation method or model training method.
[0313] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the image annotation methods or model training methods in the above embodiments.
[0314] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.
[0315] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0316] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0317] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. An image annotation method, characterized in that, Applied to electronic devices, the method includes: Display the first training image to be labeled; The system receives user annotations for the first training image and displays the bounding boxes and category labels of the user-annotated target objects on the first training image; wherein the category labels annotate at least one target object. In response to the automatic labeling command, the bounding boxes and category labels of the target objects in the second training image are displayed on the second training image; wherein, the bounding boxes and category labels of the target objects in the second training image are obtained by inputting the second training image into a target detection model trained using the labeled first training image; the labeled first training image includes: user-labeled bounding boxes and category labels.
2. The method according to claim 1, characterized in that, After receiving user annotations for the first training image and displaying the bounding boxes and category labels of the user-annotated target objects on the first training image, the method further includes: In response to the label appending instruction, the bounding box and category label of the target object in the first training image are displayed on the first training image; The bounding boxes and category labels of the target objects in the first training image are obtained by inputting the first training image into a target detection model trained using the user-annotated first training image. The labeled first training image also includes: an appended bounding box and category label.
3. The method according to claim 2, characterized in that, Before receiving user annotations for the first training image, the method further includes: In response to the label creation instruction, the system receives and saves the category labels corresponding to the target objects created by the user for the first training image.
4. The method according to claim 1 or 2, characterized in that, When the number of second training images to be labeled is greater than one, after executing the automatic labeling instruction on the first second training image, the method further includes: The next second training image to be labeled is displayed sequentially, and an automatic labeling instruction is executed for each second training image until all second training images are labeled.
5. The method according to claim 4, characterized in that, Prior to the step of displaying the first training image to be labeled, the method further includes: Obtain the training image set specified by the user; wherein the training image set includes multiple training images; Receive a training image selected by the user from the training image set as the first training image to be labeled; wherein, all other images in the training image set except the first training image to be labeled are used as the second training images to be labeled; or, The system receives a training image selected by the user from the training image set as the first training image to be labeled; and under the condition that the preset automatic labeling conditions are met, it determines at least one image from the other images in the training image set, excluding the first training image, as the second training image to be labeled.
6. The method according to claim 4, characterized in that, For the first and second training images, in response to the automatic labeling command, the bounding boxes and category labels of the target objects in the second training image are displayed on the second training image, including: In response to the automatic labeling instruction, the bounding boxes and category labels of the target objects annotated after the second training image was input into the object detection model are displayed on the second training image; and, Based on user selection, the bounding boxes and category labels of all target objects, after user correction and adjustment, are displayed on the second training image.
7. The method according to claim 6, characterized in that, After each automatic labeling instruction is executed on a second training image, the method further includes: adding the second training image, which has been corrected and adjusted by the user, to the target training set based on user selection, so as to update the target training set; After each display of the next second training image to be labeled, before responding to the automatic labeling instruction, the method further includes: training the object detection model with the updated object training set based on the user selection to update the object detection model; For the second and subsequent second training images, in response to the automatic labeling command, the bounding boxes and category labels of the target objects in the second training images are displayed on the second training images, including: Based on user selection, on the second training image, display the bounding boxes and category labels of the target objects annotated after the second training image was input into the object detection model (whether it was before or after the update); and, Based on user selection, the bounding boxes and category labels of all target objects, after user correction and adjustment, are displayed on the second training image.
8. The method according to claim 2, characterized in that, In response to the label appending instruction, the bounding box and category label of the target object in the first training image are displayed on the first training image, including: In response to the label appending instruction, the bounding box and category label of the target object to be appended are displayed on the first training image; wherein, the bounding box and category label of the target object to be appended are obtained by inputting the first training image into the target detection model trained using the first training image labeled by the user, and the similarity between the target object to be appended and the target object labeled by the user is greater than a preset threshold.
9. The method according to claim 8, characterized in that, If, after the label appending instruction is executed, the target bounding boxes and category labels of all target objects are not displayed on the first training image, the method further includes: In response to the next label appending instruction, the bounding box and category label of the target object to be appended are displayed on the first training image; wherein, the bounding box and category label of the target object to be appended are obtained by inputting the first training image into the updated target detection model, and the similarity between the target object to be appended and the already labeled target object is greater than a preset threshold. The updated object detection model is trained on a first training image containing user-annotated bounding boxes and category labels, as well as previously appended bounding boxes and category labels.
10. The method according to claim 9, characterized in that, After displaying the target bounding box and category label of the target object to be added this time, the method further includes: Based on user selection, the bounding boxes and category labels of the newly added objects have been revised and adjusted.
11. The method according to claim 3, characterized in that, Before responding to a tag creation instruction, the method further includes: displaying a tag creation control; wherein the tag creation instruction is generated by the user by clicking the tag creation control; Before receiving user annotations for the first training image, the method further includes: displaying label controls corresponding to different target objects created by the user; wherein, the bounding box and category label of each target object annotated by the user on the first training image are obtained by annotating when the user selects the label control corresponding to the target object; Before responding to the automatic marking instruction, the method further includes: displaying an automatic marking control; wherein the automatic marking instruction is generated by the user by clicking the automatic marking control; Before responding to a tag append instruction, the method further includes: displaying a smart append control; wherein the tag append instruction is generated by the user clicking the smart append control.
12. The method according to claim 11, characterized in that, The response to the automatic labeling command includes: responding to the automatic labeling command when the automatic labeling control is unlocked; wherein the unlocking condition of the automatic labeling control is: the first training image has been labeled and the second training image to be labeled is displayed; The response to the tag appending instruction includes: responding to the tag appending instruction when the smart appending control is unlocked; wherein the unlocking condition of the smart appending control is: there are user-annotated target boxes and category labels on the first training image.
13. The method according to claim 1, characterized in that, The method further includes: When a modification instruction is received for any target box, the position, size, and / or corresponding category label of the target box are modified based on the modification instruction; When a deletion instruction is received for any target box, the target box and its corresponding category label are removed from the training image; When a user adds new annotations to a training image, the user-added bounding box and category label are displayed on that training image.
14. The method according to any one of claims 1-13, characterized in that, The display of the first training image to be labeled includes: Display an interactive interface for labeling training images; the interactive interface includes: an image display area, options for adding training images, options for labeling images, and a "Train" button; After the user clicks the option to add a training image, the first training image to be labeled selected by the user is displayed in the image display area. The receiving of user annotations for the first training image includes: After the user clicks the option to annotate the image, the annotations for the first training image are received from the user. In response to the automatic labeling command, the bounding boxes and category labels of the target objects in the second training image are displayed on the second training image, including: The second training image to be labeled, selected by the user, is displayed in the image display area; An automatic labeling control is displayed on the interactive interface. In response to the automatic labeling command generated by the user clicking the automatic labeling control, the target bounding box and category label of the target object in the second training image are displayed on the second training image. The method further includes: Based on user selection, the bounding boxes and category labels of all target objects after user correction and adjustment are displayed on the second training image; After the user clicks the "Train" button, the object detection model is trained using the labeled training images to obtain the trained object detection model.
15. The method according to any one of claims 1-13, characterized in that, The first training image displays bounding boxes and category labels for different target objects annotated by the user, with each category label annotating at least one target object.
16. An image annotation method, characterized in that, Applied to electronic devices, the method includes: Display a training image that needs to be labeled; The system receives user annotations for the training image and displays the bounding boxes and category labels of the user-annotated target objects on the training image; wherein the category labels annotate at least one target object. In response to a label append instruction, the bounding box and category label of the target object in the training image are displayed on the training image; The bounding boxes and category labels of the target objects in the training images are obtained by inputting the training images into a target detection model trained using user-annotated training images.
17. The method according to claim 16, characterized in that, The number of training images to be labeled is multiple; after executing the label appending instruction for the first training image, the method further includes: The next training image to be labeled is displayed sequentially, and a label appending instruction is executed for that training image until all training images are labeled.
18. The method according to claim 16, characterized in that, After executing the label appending instruction on the first training image, the method further includes: The next training image to be labeled is displayed sequentially. In response to the automatic labeling instruction, the bounding boxes and class labels of the target objects in the training image are displayed on the next training image; wherein, the bounding boxes and class labels of the target objects in the training image are obtained by inputting the training image into the target detection model trained using the labeled training images; Return to the previous step of sequentially displaying the next training image to be labeled, until all training images have been labeled.
19. The method according to any one of claims 16-18, characterized in that, The training image displays bounding boxes and category labels for different target objects annotated by the user; wherein each category label annotates at least one target object.
20. A model training method, characterized in that, Applied to electronic devices, the method includes: Obtain the training images specified by the user; The training images are labeled using the image annotation method according to any one of claims 1-19; In response to the model training instruction, the target detection model is trained using the labeled training images to obtain the trained target detection model.
21. An image annotation device, characterized in that, Applied to electronic devices, the device includes: The display module is used to display the first training image to be labeled. The annotation receiving module is used to receive user annotations for the first training image and display the bounding boxes and category labels of the user-annotated target objects on the first training image; wherein the category labels annotate at least one target object; An automatic labeling response module is used to respond to an automatic labeling command and display the bounding boxes and category labels of the target objects in the second training image on the second training image; wherein the bounding boxes and category labels of the target objects in the second training image are obtained by inputting the second training image into a target detection model trained using a labeled first training image; the labeled first training image includes user-labeled bounding boxes and category labels.
22. An image annotation device, characterized in that, Applied to electronic devices, the device includes: The training image display module is used to display a training image to be labeled; The user annotation receiving module is used to receive user annotations for the training image and display the bounding boxes and category labels of the user-annotated target objects on the training image; wherein the category labels annotate at least one target object; The label appending instruction response module is used to display on the training image: the bounding box and category label of the target object in the training image; wherein, the bounding box and category label of the target object in the training image are obtained by inputting the training image into a target detection model trained using user-annotated training images.
23. A model training device, characterized in that, Applied to electronic devices, the device includes: The training image acquisition module is used to acquire the training images specified by the user. A training image annotation module is used to annotate the training image using the image annotation method according to any one of claims 1-19; The model training instruction response module is used to respond to model training instructions and train the object detection model using the labeled training images to obtain the trained object detection model.
24. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-15, any one of claims 16-19, or claim 20.
25. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-15, any one of claims 16-19, or claim 20.
26. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method according to any one of claims 1-15, any one of claims 16-19, or claim 20.
