Annotated-image processing method, electronic device and storage medium

Abstract

Provided are an annotated-image processing method, an electronic device and a storage medium, which relate to the technical field of computer vision, and are used for improving the efficiency of erroneous-annotation checking and ensuring the accuracy of annotation. The method comprises: acquiring a source image library, wherein the source image library comprises at least one source image and a plurality of pieces of annotation data of the at least one source image, the annotation data is used for representing annotation information which is obtained by annotating a defect in the source image, and the annotation data comprises position information (100); on the basis of respective position information of the plurality of pieces of annotation data of the at least one source image, extracting a plurality of annotated images from the at least one source image, wherein the plurality of pieces of annotation data correspond to the annotated images on a one-to-one basis (101); and displaying the plurality of annotated images corresponding to the at least one source image (102).

Description

A method for processing labeled images, an electronic device, and a storage medium. Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a method for processing labeled images, an electronic device, and a storage medium. Background Technology

[0002] Image annotation involves marking the location, category, and context of objects within an image to help computer vision models recognize and understand it. This annotation information can then be used as training data to improve the accuracy and performance of machine learning models.

[0003] Annotation software is mainly used for image annotation in the fields of machine learning and computer vision. It supports multiple annotation formats and provides a graphical user interface (GUI). Users can select annotation tools through the graphical interface and draw and add labels on the image.

[0004] Traditional annotation software can only perform manual labeling of targets on images. It is inefficient when processing a large number of images, especially when checking for annotation errors. When annotating high-resolution defect images, if the user makes an error, the defect target is relatively small and may be missed during the inspection, making it difficult to guarantee the accuracy of the annotation. Summary of the Invention

[0005] This disclosure provides a method for processing labeled images, an electronic device, and a storage medium to improve the efficiency of error labeling and ensure the accuracy of labeling.

[0006] In a first aspect, embodiments of this disclosure provide a method for processing labeled images, the method comprising:

[0007] Obtain a source image library, the source image library including at least one source image and multiple annotation data of the at least one source image, the annotation data being used to represent annotation information obtained by annotating defects in the source image, the annotation data including location information;

[0008] Based on the location information of the multiple annotation data of the at least one source image, multiple annotated images are extracted from the at least one source image, wherein the annotation data and the annotated images are in one-to-one correspondence.

[0009] Display multiple labeled images corresponding to the at least one source image.

[0010] Secondly, embodiments of this disclosure also provide an electronic device, which includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0011] Obtain a source image library, the source image library including at least one source image and multiple annotation data of the at least one source image, the annotation data being used to represent annotation information obtained by annotating defects in the source image, the annotation data including location information;

[0012] Based on the location information of the multiple annotation data of the at least one source image, multiple annotated images are extracted from the at least one source image, wherein the annotation data and the annotated images are in one-to-one correspondence.

[0013] Display multiple labeled images corresponding to the at least one source image.

[0014] Thirdly, embodiments of this disclosure also provide a processing apparatus for labeled images, the apparatus comprising:

[0015] An image acquisition module is used to acquire a source image library, the source image library including at least one source image and multiple annotation data of the at least one source image, the annotation data being used to represent annotation information obtained by annotating defects in the source image, the annotation data including location information;

[0016] An image extraction module is used to extract multiple labeled images from the at least one source image based on the location information of the multiple labeled data of the at least one source image, wherein the labeled data and the labeled images are in one-to-one correspondence.

[0017] The image display module is used to display multiple labeled images corresponding to the at least one source image.

[0018] Fourthly, embodiments of this disclosure also provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the method described in any of the first aspects above.

[0019] Fifthly, this disclosure provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects.

[0020] These or other aspects of this disclosure will become more apparent in the following description of embodiments. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is a flowchart illustrating an implementation method for processing labeled images according to an embodiment of this disclosure;

[0023] Figures 2A-2B are schematic diagrams illustrating a method for displaying annotated images according to an embodiment of this disclosure;

[0024] Figure 3A is a flowchart of an implementation method for self-checking labeled data provided in an embodiment of this disclosure;

[0025] Figure 3B is a schematic diagram of the first application interface displaying labeled images provided in an embodiment of this disclosure;

[0026] Figure 3C is a schematic diagram of the second application interface displaying labeled images provided in an embodiment of this disclosure;

[0027] Figure 4 is a schematic diagram of a third application interface provided in an embodiment of this disclosure;

[0028] Figure 5 is a schematic diagram of the dynamic loading process of an AI model service platform provided in an embodiment of this disclosure;

[0029] Figure 6 is a schematic diagram of an interface for displaying missing and / or incorrectly labeled annotation data provided in an embodiment of this disclosure;

[0030] Figure 7 is a schematic diagram of the automatic adjustment principle of auxiliary line color provided in an embodiment of this disclosure;

[0031] Figures 8A-8B are schematic diagrams comparing auxiliary lines provided in an embodiment of this disclosure;

[0032] Figures 9A-9B are schematic diagrams of a category statistics display interface provided in an embodiment of this disclosure;

[0033] Figure 10 is a schematic diagram of an image filtering interface provided in an embodiment of this disclosure;

[0034] Figure 11 is a schematic diagram of an image-level defect display interface provided in an embodiment of this disclosure;

[0035] Figures 12A-12B are schematic diagrams of a user information input function interface provided in an embodiment of this disclosure;

[0036] Figure 13 is a schematic diagram of an electronic device provided in an embodiment of this disclosure;

[0037] Figure 14 is a schematic diagram of an image processing apparatus provided in an embodiment of this disclosure. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this disclosure clearer, the disclosure will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0039] In this disclosure, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0040] The term "multithreading" in this disclosure refers to a program execution method that allows multiple tasks to be executed simultaneously.

[0041] The term "multithreading" in this disclosure refers to a program execution method that allows multiple tasks to be executed simultaneously.

[0042] The term "tree structure" in this disclosure refers to a hierarchical data organization method used for classifying and organizing data.

[0043] In this embodiment of the disclosure, the term "proportional scaling" refers to a scaling method that maintains the aspect ratio of the image.

[0044] In this disclosure, the term "annotation software" refers to a computer program that identifies and marks target regions in an image.

[0045] The application scenarios described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided in this disclosure. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided in this disclosure are also applicable to similar technical problems. In the description of this disclosure, unless otherwise stated, "multiple" means two or more.

[0046] Before introducing the image processing method for the annotated images provided in the embodiments of this disclosure, the technical background of the embodiments of this disclosure will be described in detail below for ease of understanding.

[0047] Image annotation involves marking the location, category, and scene of objects in images to help computer vision models recognize and understand them. This annotation information can serve as training data, improving the accuracy and performance of machine learning models. Annotation software is primarily used for image annotation in machine learning and computer vision. It supports multiple annotation formats and provides a graphical user interface (GUI), allowing users to select annotation tools and draw and add labels to images. For example, LabelImg is an object detection bounding box annotation software, an open-source image annotation tool in Python and Qt, mainly used for image annotation in machine learning and computer vision. Its main functions include:

[0048] (1) Image annotation: Allows users to annotate objects in images, select objects by clicking and dragging with the mouse, and add and modify labels. This is crucial for creating datasets for training machine learning models. (2) Supports multiple annotation formats: including Pascal (Pater Analysis, Statistical Modeling and Computational Learning, a network organization), VOC (Visual Object Classes, a computer vision dataset), and YOLO (You Only Look Once). Pascal VOC is an XML format, and YOLO is a text format, which is compatible with various machine learning frameworks. In addition, other annotation formats such as COCO are also supported. (3) Provides an intuitive graphical user interface (GUI): Makes the annotation process simple and convenient. Users can select annotation tools (such as rectangles, polygons, etc.) through the graphical interface and draw and add labels on the image. (4) Supports keyboard shortcuts: such as using Ctrl+N to create a new annotation and using Ctrl+S to save an annotation, further improving annotation efficiency. The annotation software solves the problem of needing a large amount of labeled image data to train models in machine learning and computer vision tasks. By providing a simple and intuitive image annotation tool, the difficulty and complexity of annotation work are effectively reduced, and annotation efficiency is improved.

[0049] However, current annotation software can only perform manual labeling of targets on images. In practical applications, when annotating high-resolution industrial defect images, if the defect target is small, user labeling errors cannot be easily and intuitively detected during inspection. Therefore, traditional annotation software can only perform manual labeling of targets on images, which is inefficient when processing large numbers of images, especially when checking for annotation errors. When annotating high-resolution defect images, if the user labels incorrectly, the small size of the defect target may cause the incorrect labeling to be missed during inspection, making it difficult to guarantee the accuracy of the annotation.

[0050] To address the shortcomings of current annotation software, this disclosure provides a method for processing annotated images. The core idea is to improve the annotation software by extracting the annotation images corresponding to the annotation data from the source image, thereby displaying the annotated area image in the source image, i.e., the annotation image. Since the annotation image is extracted from the source image, displaying the annotation image allows inspectors to more intuitively observe the annotation images of each annotation data in the source image. Especially when annotating defects in source images in the industrial field, the annotation images corresponding to the annotated defects can be displayed more intuitively, making it easier for inspectors to quickly find the incorrectly annotated data from the displayed annotation images corresponding to the annotation data, thereby more quickly inspecting the annotated defect data and ensuring the accuracy of the annotation data.

[0051] This embodiment provides a method for processing labeled images. This method is applied to an electronic device and can be implemented through a target application program on the electronic device. Optionally, in addition to implementing the image processing method of this disclosure, the target application program can also implement image annotation functions, which is equivalent to the functions of annotation software. This embodiment can also improve the annotation software based on its original image annotation functions, adding the functions implemented by the image processing method of this disclosure. Thus, it can not only annotate images but also quickly view and verify the annotation data, ensuring the accuracy of the annotation data.

[0052] As shown in Figure 1, the implementation flow of the image annotation processing method provided in this embodiment is as follows:

[0053] Step 100: Obtain a source image library, which includes at least one source image and multiple annotation data of the at least one source image. The annotation data is used to represent annotation information obtained by annotating defects in the source image, and the annotation data includes location information.

[0054] The source image library includes one or more source images. The types of source images are distinguished based on their content; for example, different products may have different types of source images. A source image includes one or more source images, and these source images are bound to annotation data. That is, at least one source image in the source image library corresponds to annotation data, and the annotation data for each source image includes annotation data for at least one category. The source image library may contain unannotated source images, i.e., images where no defects were found. These unannotated source images do not have bound annotation data. The source image library may also contain annotation data errors, omissions, etc.

[0055] Optionally, in this embodiment, at least one source image corresponds to multiple annotation data. The multiple annotation data in this embodiment correspond to one source image or multiple source images; that is, the multiple annotation data may be multiple annotation data within one source image or multiple annotation data within multiple source images. The multiple annotation data may belong to one category or multiple categories. In this embodiment, the multiple annotation data may be multiple annotation data under one category within one source image, multiple annotation data under multiple categories within one source image, multiple annotation data under one category within multiple source images, or multiple annotation data under multiple categories within multiple source images. This embodiment does not impose excessive limitations on this.

[0056] In implementation, the source image library in this embodiment refers to the storage space that stores source images and their annotation data, such as in the form of a database. Obtaining the source image library in this embodiment refers to the process of reading source images and their annotation data from the source image library, rather than directly obtaining the database.

[0057] It should be noted that, in this embodiment, "defect" refers to abnormal pixels in the source image, which correspond to product defects in the actual product. The source image can represent an image of the actual product. In industrial defect detection, defects can be considered as anomalies in the product's appearance. To detect defective products, a source image of the product can be acquired, and abnormal pixels (defects) in the source image can be detected. The detection results are used to determine whether the product has a defect. This embodiment does not impose excessive limitations on the specific definition, shape, size, or other information regarding defects.

[0058] Optionally, the annotation data in this embodiment is used to represent annotation information obtained by annotating defects in the source image. Optionally, the annotation data includes, but is not limited to, at least one of the following: defect category in the source image, defect location (i.e., the position of the annotation box), defect size (i.e., the size of the annotation box), and defect coordinates (i.e., the coordinate information of the annotation box). Optionally, the annotation data may also include other custom information, such as user data, which is bound to the annotation data to facilitate backtracking and determining the user who made the annotation when an error is detected, thereby facilitating timely adjustment of the annotation data by that user.

[0059] Optionally, the source image also includes the annotation user information corresponding to each annotation data, so that when erroneous samples are traced back, the relevant annotation personnel can be found and adjustments can be made in a timely manner.

[0060] In some embodiments, the annotation data of the source image can be obtained through any one or more of the following methods:

[0061] Method 1a) In response to a user's annotation operation on defects in the source image in the target application, obtain the annotation data of the source image;

[0062] In implementation, the target application in this embodiment has an image annotation function, allowing users to annotate source images and obtain annotation data. The target application can also extract and display the annotated images corresponding to the annotation data from the source images. Here, defects in the source image refer to abnormal pixels.

[0063] Optionally, it can also retrieve annotation data of the source image in response to a user's annotation operation on the target webpage regarding defects in the source image.

[0064] In practice, the target webpage has an image annotation function, allowing users to annotate source images on the webpage and obtain annotation data for the source images. The target webpage can also display the annotated images.

[0065] Method 1b) Obtain the first detection result of the detection model on the source image for defect detection, and determine the annotation data of the source image based on the first detection result.

[0066] In implementation, a detection model can be used to detect defects in the source image, automatically labeling the defects in the source image to obtain a first detection result. This first detection result is the first annotation data obtained by automatically labeling the source image, and thus the first annotation data is determined as the annotation data of the source image. Specifically, the first detection result of the detection model on the source image is obtained through the target application, and the target application determines the annotation data of the source image based on the first detection result.

[0067] Optionally, the first detection result in this embodiment includes, but is not limited to, the first annotation data, which includes, but is not limited to, at least one of the following: defect category in the source image, defect location (i.e., the position of the annotation box), defect size (i.e., the size of the annotation box), and defect coordinates (i.e., the coordinate information of the annotation box).

[0068] Optionally, the annotation data includes location information, which includes, but is not limited to, at least one of the following: the position of the annotation box, the coordinate information of the annotation box (such as the coordinates of the four corners of the annotation box), and the size of the annotation box.

[0069] Optionally, defect detection of the source image can be achieved through a designated application other than the target application on the electronic device. This designated application can provide AI defect detection services. The application stores one or more detection models and loads the one or more detection models. The loaded detection models are used to automatically annotate the source image, thereby sending the first detection result to the target application. The target application determines the annotation data of the source image based on the annotation data in the first detection result, and then determines the source image library containing the source image and the annotation data of the source image.

[0070] Optionally, defect detection of the source image can be implemented on the server side. The server provides one or more detection models, loads one or more detection models, and uses the loaded detection models to automatically annotate the source image, thereby sending the first detection result to the target application of the electronic device. The target application combines the first annotation data in the detection result with the source image to obtain the source image library.

[0071] It should be noted that the source images in the source image library in this embodiment can also be obtained in other ways, such as through the USB interface of an electronic device, WIFI, Bluetooth, etc. This embodiment does not impose too many limitations on this.

[0072] Step 101: Based on the location information of the multiple annotation data of the at least one source image, extract multiple annotated images from the at least one source image, wherein the annotation data and the annotated images are in one-to-one correspondence;

[0073] In this context, there is a one-to-one correspondence between labeled data and labeled images. A labeled image is an image of a portion of a source image extracted from that source image. A source image contains labeled data of one or more categories.

[0074] In some embodiments, the annotation data includes location information, which includes, but is not limited to, at least one of the following: defect location (i.e., the position of the annotation box), defect size (i.e., the size of the annotation box), and defect coordinates (i.e., the coordinate information of the annotation box) in the source image. The annotated image corresponding to the annotation data is extracted from the source image in the following manner:

[0075] Based on the location of the annotation data in the source image, the source image is cropped to obtain an annotated image corresponding to at least one category of annotation data.

[0076] In implementation, the location of the annotation data in the source image is determined. The annotation data includes bounding boxes; that is, based on the location of the bounding box in the source image, the region of the source image corresponding to that bounding box is cropped to obtain the annotation image corresponding to that bounding box. The category of the annotation image is determined based on the category of the annotation data.

[0077] In some embodiments, the source image is cropped in the following manner:

[0078] The number of threads is determined based on the current number of CPUs (Central Processing Units); when the number of threads is greater than 1, multiple threads are started to crop the source image in parallel.

[0079] In implementation, this embodiment can determine the number of threads based on the number of CPUs in the current electronic device. There can be a one-to-one correspondence between CPUs and threads, meaning one CPU corresponds to one running thread. Thus, when the electronic device has multiple CPUs, multiple threads can be used to perform the cropping process of the source image in parallel, improving image data processing efficiency.

[0080] In some embodiments, since the defect targets annotated in the source image are usually quite small, in order to ensure that the displayed annotated image is easier to observe, this disclosure may further perform the following steps after cropping the source image and before obtaining the annotated image corresponding to at least one category of annotation data, to ensure the visibility of the annotated image, as follows:

[0081] The source image is cropped to obtain a cropped image corresponding to at least one category of labeled data; the cropped image is scaled proportionally, and the scaled image is determined as the labeled image; wherein the size of the longest side of the scaled cropped image is within a preset range.

[0082] In some embodiments, multiple labeled images are extracted from the at least one source image based on the location information of each of the multiple labeled data of the at least one source image in the following manner:

[0083] Based on the position information of each of the multiple annotation data of the source image, the source image is cropped to obtain cropped images corresponding to the multiple annotation data; the multiple cropped images are scaled proportionally to obtain the multiple annotation images; wherein the size of the longest side of the scaled cropped image is within a preset range.

[0084] In practice, to ensure the visibility of the labeled images, when there is too much labeled data, resulting in too many labeled images and excessive memory usage, this disclosure will scale the cropped images. To ensure that the scaled images do not affect the inspection of defects in the labeled images, proportional scaling can be used, and the size of the longest side after scaling is between 200 and 300 pixels (inclusive). This ensures that defects in the scaled labeled images will not be distorted and will not affect the inspection and verification of the defects.

[0085] Step 102: Display multiple labeled images corresponding to the at least one source image.

[0086] Optionally, the source image in this embodiment includes labeled images corresponding to at least one category of labeled data, wherein the labeled data under one category includes one or more, that is, the labeled images under one category can be one or more.

[0087] In some embodiments, the labeled image corresponding to the source image may be displayed in any one or more of the following ways:

[0088] Method 2a) Display the labeled images corresponding to the labeled data according to the category of the labeled data in the source image.

[0089] This method displays labeled images according to different categories, meaning it shows labeled images of different categories separately. Users can select to display labeled images from one category or multiple categories. This allows users to view all labeled images under a category, thus enabling them to more quickly identify labeled images with errors, i.e., to quickly locate labeled images that do not belong to the same category from among labeled images of the same category.

[0090] In some embodiments, labeled images corresponding to different categories of labeled data of the source image are displayed in a tree structure, wherein a root node in the tree structure represents labeled data of one category. When a user clicks on a root node, the labeled data of the category corresponding to that root node is displayed.

[0091] Method 2b) Display the labeled images corresponding to the labeled data of each category in each source image according to different source images.

[0092] This method displays labeled images according to the source image, meaning that for each source image, all labeled images of all categories within that source image are displayed. This allows users to more intuitively view and verify the labeled images of each category in the source image, and facilitates the identification of erroneous labeled images by comparing different categories of labeled images.

[0093] In practice, the system can display labeled images under each category corresponding to the source image, or it can categorize and display labeled images under each category separately. By providing different display levels, it offers users a more comprehensive viewing experience, facilitating observation of labeled images from different perspectives and improving labeling efficiency.

[0094] In some embodiments, this embodiment displays the labeled image corresponding to the source image through the following steps:

[0095] Step 1) Determine the target memory size required to display multiple labeled images corresponding to the at least one source image based on the number and size of the labeled images;

[0096] Optionally, the target memory size can be determined as follows:

[0097] Based on the number and size of the labeled images, determine the total runtime memory required to display the labeled images corresponding to each category of labeled data in the at least one source image; and determine the total runtime memory as the target memory size.

[0098] In practice, the total data and total size of the labeled images corresponding to each source image in the source image library are statistically analyzed. The total running memory size required for the labeled images corresponding to all categories of labeled data for each source image in the source image library is determined, and this total running memory size is determined as the target memory size required for multiple labeled images.

[0099] Memory is one of the most important components of a computer, used for temporary storage of data and programs. It acts as a bridge for interaction with the CPU, carrying most of the data and programs used during computer operation. The primary function of memory is to provide temporary storage for the operating system and various programs while the computer is running.

[0100] Step 2) Based on the target memory size and the available physical memory of the target application, save the multiple labeled images to the specified storage location;

[0101] During implementation, the target memory size and the available physical memory size are compared to determine whether to save multiple labeled images to physical memory or local disk.

[0102] Physical memory is used in contrast to virtual memory. Physical memory refers to the memory space obtained through physical memory modules. Common physical memory specifications include 256MB, 512MB, 1GB, and 2GB. With the development of computer hardware, 4GB, 8GB, and even higher capacities have emerged. When physical memory is insufficient, virtual memory can be used as a substitute. Physically, physical memory refers to the actual capacity of the memory modules inserted into the motherboard's memory slots. In this embodiment, available physical memory refers to the remaining (unoccupied) usable physical memory in the current electronic device or server. The available physical memory for the target application refers to the physical memory that can be allocated from the available physical memory in the current electronic device or server, allowing the target application to run normally.

[0103] In some embodiments, the specified storage location for saving the labeled image is determined in the following manner:

[0104] (1) If the available physical memory meets the target memory size, then the plurality of labeled images are loaded into the physical memory of the target application;

[0105] Optionally, after the labeled images are loaded into physical memory, when displaying the loaded labeled images on the target application, the labeled images for each category can be displayed separately. This allows users to more intuitively view the labeled images of defects in each category, thereby locating incorrectly labeled defects more quickly.

[0106] In practice, the labeled images can be loaded directly into the physical memory of the target application, and the labeled images can be displayed directly in the interface of the target application. One possible implementation is that after the annotator annotates the source image in the target application, not only can the entire annotated source image be displayed, but each annotated defect can also be displayed in the target application through the labeled image. This makes it convenient for users to not only see the annotation of defects in the entire source image, but also to view the labeled images of each category of defects for each annotated defect.

[0107] (2) If the available physical memory does not meet the target memory size, the multiple labeled images are saved to the local disk.

[0108] In some embodiments, the specified storage location includes a local disk; according to the category of the annotation data in the source image, the annotation image corresponding to each category of annotation data is saved to the same directory on the local disk.

[0109] In practice, when the target application's available physical memory is insufficient, the image can be directly saved to the local directory on the local disk. Users can still view the labeled images in the local directory by checking the local resource manager. Furthermore, labeled images of different categories can be stored by category, and labeled images of different categories can be saved to different directories by category, while labeled images of the same category can be saved to the same directory. This makes it convenient for users to view labeled images of defects by category, thereby facilitating the quick location of labeled images of defects that do not belong to that category from labeled images of the same category.

[0110] In some embodiments, the storage name of the labeled image on the local disk is determined based on at least one of the source image name, the category of the labeled data, and the location information of the labeled data.

[0111] In practice, by defining the storage name in the local directory under the labeled image, labeled images of different source images, different categories, and different defect locations can be distinguished. The name of the labeled image can determine at least one of the following: the source image to which the labeled image belongs, the defect category, and the location information of the defect in the source image. This facilitates the establishment of the association between labeled images, source images, categories, and location information, and allows for faster retrieval of relevant information for labeled images, greatly facilitating problem tracing and searching.

[0112] Step 3) Based on the specified storage location, display multiple labeled images corresponding to the at least one source image.

[0113] In some embodiments, the labeled image is displayed in one or more of the following ways:

[0114] Method 1) Load the labeled image into the physical memory of the target application;

[0115] In this method, the multiple labeled images are displayed on the first application interface of the target application. Alternatively, the source image containing labeled data (label boxes) can also be displayed on the first application interface.

[0116] In implementation, when sufficient physical memory is available, the labeled images can be directly loaded into physical memory. The target application internally records each labeled image and its corresponding position in the source image. Users can then directly click on the labeled image with the problem to jump to the source image's display interface. This interface displays the complete source image, along with the labeled data (label boxes), at the labeled locations within the source image. Furthermore, the labeled boxes can be highlighted to emphasize the labeled images with problems that the user wants to view. For example, when a user clicks on a labeled image, the corresponding labeled box (label data) in the source image will be highlighted, making it easier for the user to locate and modify it.

[0117] Method 2) Save the labeled image to the local disk;

[0118] In this approach, one scenario involves displaying the multiple labeled images in a local resource management interface. For example, users can view the labeled images in folders within their local directory. Optionally, labeled images can also be saved by category; for instance, labeled images of different categories can be saved in different folders within the local directory. The folder name identifies the corresponding category, making it easier for users to view labeled images under different categories and more quickly identify any labeled errors.

[0119] Another approach is to display the multiple labeled images in the local resource management interface; in response to a user dragging and dropping the labeled images to the target application, the multiple labeled images are displayed in the second application interface of the target application. For example, if labeled images are displayed in a folder in the local directory, and the user drags a labeled image that they deem problematic to the application interface of the target application, the labeled image will be displayed in the application interface of the target application. Optionally, the user can also drag and drop only one labeled image to the second application interface of the target application for display, or the user can drag and drop multiple labeled images to the second application interface of the target application for display.

[0120] During implementation, when available physical memory is insufficient, each labeled image is saved to a local directory according to its category. Users can select the labeled images that seem problematic in the local file explorer and drag them into the second application interface for viewing.

[0121] In some embodiments, this embodiment may also provide a defect viewing function, the specific implementation process of which is as follows:

[0122] Process (1) responds to the user's viewing operation of the displayed labeled image, determines the location information of the labeled image, and jumps to the display interface of the source image corresponding to the labeled image;

[0123] In some embodiments, location information is determined in the following manner:

[0124] The target application is used to record the correspondence between each labeled image and the position information of each labeled image in the source image; based on the correspondence, the position information of the labeled image in the source image is determined.

[0125] In practice, the labeled image is cropped from the source image based on the labeled data. The position information of the labeled image in the source image can be recorded, thereby establishing a correspondence between the position information of the labeled image and the source image. Using the correspondence in the target application, the position information of the labeled image in the source image can be determined.

[0126] In some embodiments, the labeled image is stored on a local disk; the location information of the labeled image in the source image is determined according to the storage name of the labeled image in the local directory of the local disk.

[0127] In implementation, when annotated images are saved to a local directory on a local disk, the location information of the annotated image can be included in the storage name of the saved annotated image. For example, the naming rule for annotated images can be designed as: source image name + location information of annotation data + category of annotation data + ".jpg". The target application can use the storage name of the annotated image to trigger a jump to the display interface of the source image corresponding to the annotated image when the user wants to view the information of the annotated image in the source image by clicking or other means. For example, when the user clicks on the annotated image in the target application, it triggers the recognition of the storage name of the annotated image, thereby matching the location of the annotation data corresponding to the annotated image in the source image, and jumping to the display interface of the source image. The display interface displays the source image and the annotation data of the source image. It can also distinguish and display the annotation data (annotation box) corresponding to the user's viewing operation.

[0128] Optionally, viewing operations include, but are not limited to, any one of: single-click, double-click, right-click, and drag-and-drop. This embodiment does not impose excessive limitations on these operations. In implementation, when a user sees an annotated image on the target application's display interface, the user can click on the annotated image to trigger the acquisition of the annotated image's location information, display the annotation data on the source image's display interface, and highlight the annotation data corresponding to the annotated image clicked by the user.

[0129] Process (2) Based on the location information, display the source image corresponding to the labeled image and the labeling data corresponding to the labeled image on the display interface.

[0130] In practice, the location of the annotation data in the source image is determined based on the location information. Thus, when displaying the source image and the annotation data, the annotation data can be highlighted or displayed separately. That is, only the annotation data corresponding to the annotation image specified by the user's viewing operation is displayed on the source image.

[0131] In some embodiments, this embodiment may also display the source image to which the labeled image corresponding to the viewing operation belongs, and the labeling data of the source image, on the display interface, and distinguish and display the labeling data corresponding to the viewing operation; or, display the source image to which the labeled image corresponding to the viewing operation belongs, and the labeling data corresponding to the viewing operation, on the display interface.

[0132] In practice, when a user finds a problem with an annotated image, they can perform a "view" operation on the annotated image. This triggers the display of the corresponding source image and the annotation data (annotation boxes) within the source image. The annotation data corresponding to the annotated image indicated by the "view" operation is highlighted, making it easier for the user to locate the problematic annotation data within the source image.

[0133] In some embodiments, different categories of annotation data in the source image can also be displayed using different display formats. For example, the annotation boxes for category A are red, while the annotation boxes for category B are blue. Optionally, the display format includes, but is not limited to, line style, line thickness, and line color.

[0134] This embodiment provides a self-checking function for annotation categories. This function can crop out all annotated images with annotation defects from the source image, allowing for a more intuitive view of whether there are annotation errors in each category on a single display interface. The specific implementation process is as follows: the target application determines the number of threads based on the current CPU count and starts multi-threaded parallel cropping. The cropped annotated images for each category are displayed in a tree structure. Clicking on different root nodes allows viewing all annotated images under that root node's category. The advantage of this is that it's easy to see at a glance whether there are annotation errors, i.e., incorrect annotation categories. For example, if an annotated image of a car appears in a group of annotated images categorized as "humans," the car's annotated image will be very prominent in the "humans" category display interface, facilitating quick location of the erroneous annotation data. As shown in Figures 2A-2B, this embodiment provides a method for displaying annotated images. Referring to Figure 2A, NG_A, NG_B, NG_C, and NG_D represent the categories of annotation data in the same source image. The source image's annotation data includes four categories, and six annotated images are displayed under category NG_B. Users can quickly locate whether these six annotated images contain annotation errors. Referring to Figure 2B, when a user thinks that labeled image 2 may be labeled incorrectly, they can double-click on labeled image 2 to trigger a jump to the display interface of the source image to which labeled image 2 belongs, and display the source image and the source image's label data (label boxes), and highlight the label box of labeled image 2.

[0135] As shown in Figure 3A, this embodiment also provides a flowchart of a method for self-checking labeled data. The specific implementation steps of this process are as follows:

[0136] Step 300: Obtain a source image library containing the source image and its annotation data;

[0137] Step 301: Determine the total number of labeled images that need to be cropped based on the labeled data;

[0138] Step 302: Determine the number of threads based on the number of CPUs in this device. When the number of threads is greater than 1, start multiple threads to crop the source image in parallel.

[0139] Step 303: Determine the target memory size required for the labeled images based on the number and size of the labeled images;

[0140] Step 304: Determine whether the available physical memory of the target application can meet the target memory size. If yes, proceed to step 305; otherwise, proceed to step 306.

[0141] Step 305: Load the labeled images into the physical memory of the target application, and display the labeled images corresponding to each category of labeled data in the first application interface according to the category of labeled data. Then execute step 308.

[0142] Referring to Figure 3B, the provided first application interface is a schematic diagram of the labeled images, which are displayed according to different categories.

[0143] Step 306: According to the category of labeled data, save the labeled images corresponding to each category of labeled data to the same directory on the local disk;

[0144] The storage name of the labeled image is determined based on at least one of the following: the name of the source image, the category of the labeled data, and the location information of the labeled data.

[0145] Step 307: Display the labeled image in the local resource management interface. In response to the user dragging and dropping the labeled image to the target application, display the labeled image in the second application interface of the target application.

[0146] Referring to Figure 3C, a schematic diagram of a second application interface for displaying labeled images is provided. When a user drags and drops labeled images from a local folder to the second application interface of the target application, each labeled image dragged by the user is displayed sequentially on the second application interface.

[0147] Step 308: In response to the user's viewing operation of the displayed annotation image, obtain the location information of the annotation image, and jump to the display interface of the source image. Based on the location information, display the source image and highlight the annotation data corresponding to the annotation image on the display interface.

[0148] The quality of dataset annotation is crucial for improving AI accuracy. This embodiment enhances the quality of user-annotated datasets, reducing unnecessary mislabeling and omissions. It also improves annotation efficiency through a one-click annotation category self-check function, allowing for quick identification and centralized review of annotation errors, significantly increasing efficiency. This one-click annotation category self-check function is an interactive interface that allows users to quickly check for errors in annotations. Through multi-threaded parallel processing and a tree-like organizational structure, it visually displays annotation defects for each category and highlights annotation boxes for inspection, enabling users to intuitively identify and correct errors. This function supports local storage solutions when memory is insufficient and provides image scaling to optimize memory usage.

[0149] In some embodiments, this embodiment also provides a function for AI verification of labeled data, and the specific implementation process is as follows:

[0150] Obtain the second detection result of the detection model for defect detection of the source image, and verify the accuracy of the annotation data of the source image based on the second detection result.

[0151] Optionally, the detection model in this embodiment can be deployed on a server or on the local electronic device (machine). This embodiment does not impose excessive limitations on this. Regardless of where the detection model is deployed, the second detection result of the detection model can be obtained on the local electronic device, and the accuracy of the labeled data can be verified based on the second detection result.

[0152] Optionally, the detection model in this embodiment is an AI (artificial intelligence) model. It can be trained using source sample images obtained by annotating the source image using a target application or annotation software, or it can be trained using other sample images. This embodiment does not impose excessive limitations on this. The detection model in this embodiment is used for defect detection. The detection model can be an existing model or a custom model. As long as it can achieve the defect detection function, it falls within the protection scope of this disclosure.

[0153] Optionally, this embodiment provides an AI model service platform for acquiring and loading a detection model to send detection results to a target application. The detection results include a first detection result and a second detection result. This AI model service platform can be understood as an application that can establish a connection with the target application to achieve data interaction. The AI ​​model service platform can be deployed on a server or on the electronic device itself; that is, the electronic device in this embodiment can deploy both the target application and the AI ​​model service platform. This embodiment does not impose further limitations on this aspect.

[0154] In some embodiments, the second detection result of the detection model performing defect detection on the source image can be obtained in the following manner:

[0155] In response to the API interface address information entered by the user in the third application interface of the target application, the second detection result of the detection model performing defect detection on the source image is obtained.

[0156] As shown in Figure 4, this embodiment also provides a schematic diagram of a third application interface. The user can configure the API interface address in the third application interface of the target application. The target application sends a model loading request to the AI ​​model service platform, carrying source image information, such as the storage address information of the source image, in the model loading request. The AI ​​model service platform loads the detection model requested by the target application, obtains the source image according to the source image information, uses the detection model to perform defect detection on the source image to obtain a second detection result (second annotation data), and returns the second detection result to the target application. The target application automatically updates the annotation data corresponding to the current source image based on the second annotation data.

[0157] In some embodiments, defect detection of the source image is performed using a detection model through the following process:

[0158] Process a) Obtain a detection model that meets the custom API interface standard;

[0159] Optionally, this embodiment provides an example of a custom API interface standard, as shown below:

[0160] (1) Load the model function and give the model a unique name;

[0161] (2) Input image data for detection and recognition, and return results in the following JSON format:

[0162] (3) Unloading model.

[0163] In practice, when the detection model meets the above API interface standards, it can be directly connected to the AI ​​model service platform, which provides services such as:

[0164] The API service is located at http: / / {IP address of the service launcher}:6060 / api / detect / {custom model name}. The API service only needs to obtain the IP address and name of the detection model to automatically load it.

[0165] Process b: Dynamically load the detection model based on the current hardware resources, and use the loaded detection model to perform defect detection on the source image.

[0166] In practice, when there are too many detection models and the deployment end (server or this electronic device) of the AI ​​model service platform cannot load enough detection models, this embodiment can also dynamically load detection models according to the hardware resources of the current electronic device to ensure that the automatic annotation requirements of the target application are met.

[0167] Optionally, the hardware resources in this embodiment include, but are not limited to, at least one of memory, video memory, and CPU.

[0168] In some embodiments, the detection model is dynamically loaded based on current hardware resources in the following manner:

[0169] In response to a model loading request from a target application, based on current hardware resources and the number of currently loaded detection models, determine whether it is necessary to unload already loaded detection models before loading the detection model requested by the target application.

[0170] During implementation, when the AI ​​model service platform starts, the detection models are loaded sequentially according to the order stored on the platform. Users can also customize the loading order of the detection models. Loading stops when the current hardware resources are limited and no more detection models can be loaded. At this point, the identification information of the loaded and unloaded detection models is recorded. The identification information includes at least one of the following: name, identifier, and code.

[0171] In some embodiments, the determination of whether to unload the loaded detection model is made in the following manner:

[0172] If the number of currently loaded detection models reaches a threshold, the detection model with the fewest requests among the loaded detection models will be unloaded, and the detection model requested by the target application will be loaded. The threshold is determined based on the current hardware resources, and the number of requests indicates the number of times the detection model is requested to be loaded by the target application.

[0173] Optionally, the loading timing of the detection models in this embodiment includes, but is not limited to, loading each detection model when the AI ​​model service platform starts, and / or loading the requested detection model when the AI ​​model service platform receives a model loading request sent by the target application.

[0174] In practice, when the target application needs to use the AI ​​model service platform to automatically annotate the source image, it will send an HTTP request (i.e., a model loading request) to the AI ​​model service platform to request the AI ​​model service platform to use the requested detection model to automatically annotate the source image. The AI ​​model service platform will count the number of requests for each loaded detection model and sort them from high to low or low to high. If the currently requested detection model has not yet been loaded, it can unload the detection model with the fewest requests among the loaded detection models and load the detection model requested by the current target application.

[0175] Optionally, this threshold is determined based on the hardware resources of the AI ​​model service platform, that is, the maximum number of detection models that can be loaded is calculated based on the hardware resources, and the threshold is determined based on this maximum number. The AI ​​model service platform can be a server or an electronic device where the target application resides; this embodiment does not impose further limitations on this.

[0176] As shown in Figure 5, this embodiment provides a flowchart of the dynamic loading process of an AI model service platform. The AI ​​model service platform connects to one or more detection models, each of which meets a custom API interface standard. The AI ​​model service platform determines the maximum number of detection models that can be loaded based on its local hardware resources, for example, a maximum of 3. The AI ​​model service platform has already loaded detection models A, B, and C, but has not yet loaded detection models D and N. The AI ​​model service platform only marks the unloaded detection models. It also records the number of times each detection model is requested to be loaded by the target application. When the AI ​​model service platform receives a model loading request (HTTP interface request) for detection model N from the target application, it determines that detection model N has not yet been loaded and the current loading count has reached the threshold. At this time, based on the number of times each loaded detection model has been requested, it unloads detection model C, which has been requested the least, and loads the requested detection model N. Then, it calls detection model N to automatically annotate the source image and returns the second detection result to the target application.

[0177] This embodiment allows configuring the HTTP API interface service address of the detection model provided by the AI ​​model service platform to be called on the target application, thereby enabling the function of automatically annotating or annotating all specified source images with AI. The target application parses the output content (JSON content) of the detection model returned by the AI ​​model service platform and automatically updates the annotation information of the specified source images in the target application.

[0178] After loading the detection model requested by the target application, the AI ​​model service platform first parses the source image information sent by the target application to obtain the source image data. Then, it calls the detection model to automatically annotate defects, obtaining second annotation data. This second annotation data is packaged and returned to the target application. The target application uses the second annotation data to correct errors in the annotation data of the source image. Specifically, the target application can compare the annotated data in the source image with the automatically annotated second annotation data on the display interface, thereby helping to remind the user to check for inconsistencies between the second and third annotation data and to confirm whether there are any omissions or errors in annotation.

[0179] In some embodiments, the accuracy of the annotation data of the source image is verified based on the second detection result in the following manner:

[0180] Determine the matching degree between the annotation content corresponding to the second detection result and the annotation image corresponding to the annotation data; determine whether there is missing and / or incorrectly labeled annotation data in the source image based on the matching degree.

[0181] In implementation, the source image is automatically annotated using a detection model to obtain a second detection result (i.e., second annotated data). The IoU (Intersection over Union) algorithm is used to determine the matching degree between the annotated content corresponding to the second detection result (i.e., the defect image corresponding to the second annotated data) and the annotated image corresponding to the annotated data. Based on the matching degree, the positional relationship between the second annotated data and the annotated data, as well as the image content, is compared to determine whether there are any missing or incorrectly annotated data. Here, IoU refers to the intersection-over-union ratio between bounding boxes (i.e., annotation boxes); the larger the value, the higher the overlap ratio. It should be noted that the matching degree includes the degree of matching between the positional information of the second annotated data (second annotation box) and the annotated data (annotation box), as well as the degree of matching between the image content of the defect image corresponding to the second annotated data and the annotated image.

[0182] This embodiment provides an annotation correction function, which is based on the HTTP API interface service provided by the AI ​​model service platform. For source images that have already been annotated, automatic annotation can be performed through a detection model, and the annotation data of the annotated source image is checked based on the second detection result (second annotation data) output by the detection model. Since the source image has a high resolution, users may not have viewed the entire image, resulting in missed annotations. In addition, it is possible that the annotator has labeled the wrong category without noticing. In this case, this function is designed with two auxiliary reminder models: one is to remind the user of the missing annotation data when the source image is detected to have missing annotations; the other is to remind the user of the incorrect annotation data when the source image is detected to have incorrect annotations.

[0183] Optionally, missing and / or incorrectly labeled data can be identified using the following methods:

[0184] a) If the matching degree between the defect image corresponding to the second detection result and the annotation image corresponding to the annotation data in the location matching second detection result and annotation data is greater than the matching threshold, it means that the defect image and the annotation image are matched and the annotation data is accurate.

[0185] b) If the degree of matching between the defect image corresponding to the second detection result and the annotation image corresponding to the annotation data is less than or equal to the matching threshold, it indicates that the defect image and the annotation image do not match, and the category of the annotation data may be incorrect.

[0186] c) If there is no data in the labeled data that matches the position of the second detection result, it indicates that there may be a case of omission in labeling, that is, the defect image corresponding to the second detection result at that position may have been omitted from labeling.

[0187] In some embodiments, missing and / or incorrect annotation data may also be displayed in the source image; wherein the display formats of the missing and incorrect annotation data are different.

[0188] In practice, when it is determined that there are missing and / or incorrectly labeled annotation data in the source image, the missing and incorrectly labeled annotation data can be displayed separately using different display formats.

[0189] Optionally, the display format includes, but is not limited to, at least one or more of the following: color of the annotation box, line shape, line thickness, shape of the annotation box, and transparency of the annotation box.

[0190] In one possible example, this embodiment can use the IoU algorithm to match the second detection result (second annotation data) with the already annotated annotation data to determine whether there are any missing or incorrectly annotated annotation data. As shown in Figure 6, this embodiment provides an interface diagram for displaying missing and / or incorrectly annotated annotation data. The target application first confirms whether there is any missing image data based on the matching degree of the position information of the second detection result and the annotation data. If so, as shown in Figure 6, a flashing semi-transparent circular area of ​​a first specified color will be marked in the source image displayed on the target application's display interface to remind the user that there may be a missing annotation. For IoU-matched annotation data, that is, when the position information and image data of the second detection result (second annotation data) and the already annotated annotation data match, their respective categories are determined, that is, whether the categories of the second detection result and the annotation data are consistent. If they are inconsistent, a flashing semi-transparent circular area of ​​a second specified color will be marked in the source image displayed on the target application's display interface to remind the user that there may be an annotation error. The user can intuitively determine whether there is a mislabeling or a missing annotation by the color of the circular area.

[0191] Optionally, this embodiment can choose to use any one of a circular region, a triangular region, or a hexagonal region to label the image data with missing or incorrect labels, thereby distinguishing it from the rectangular boxes used for labeling data. Furthermore, to more intuitively reflect the size of the missing label region, the diameter of the circular region can be taken as the smallest outer circle of the rectangular box (i.e., the labeling box) in the second detection result output by the detection model. Optionally, the transparency of the region of the missing and / or incorrect labeling box can also be set.

[0192] The annotation correction function provided in this embodiment is mainly used to remind annotators of possible suggestions. It will not directly modify the annotation information. Whether or not modification is needed ultimately depends on the annotator's judgment, thereby fully ensuring the accuracy of the annotation data.

[0193] This embodiment enhances annotation accuracy by utilizing an AI model service platform and AI automatic annotation function to reduce human error and improve annotation accuracy. It supports user-defined algorithm models, enabling plug-and-play functionality via an HTTP interface. The AI ​​model service platform provides HTTP service interfaces for loading detection models, image detection and recognition, and unloading detection models, supporting the integration and application of custom detection models. This embodiment also features a dynamic loading mechanism, optimizing the performance of multi-model calls to address hardware resource limitations. It optimizes memory management, effectively processing large amounts of image data even with insufficient physical memory through intelligent memory management strategies. Users can define their own AI models and call them via the HTTP service interface to achieve AI-assisted annotation and error correction functions. The platform incorporates a dynamic model loading and unloading mechanism to adapt to different hardware resource conditions. Utilizing the AI ​​model service platform, it achieves intelligent detection of automatic and erroneous annotations, alerting users to potential omissions or errors through flashing, semi-transparent circular areas of a specified color. This embodiment also includes a dynamic memory and storage switching mechanism to ensure efficient operation under different hardware conditions. This embodiment can improve the reliability of labeled data. The labeling correction function serves as a reminder rather than automatic modification, ensuring that the final accuracy of the labeled data is determined by humans, thus enhancing the reliability of the data.

[0194] In some embodiments, this embodiment also provides a scheme for automatic adjustment of annotation guide lines, which makes the guide lines more obvious and improves annotation efficiency when annotating. The specific implementation process is as follows:

[0195] Display auxiliary lines for annotation; adjust the color of the auxiliary lines based on the pixels in the adjacent area at the cursor position.

[0196] In implementation, when entering the annotation interface, auxiliary lines are displayed in real time as the cursor moves. The position of the auxiliary lines moves with the cursor position. This embodiment automatically adjusts the color of the auxiliary lines based on the pixels in the adjacent areas of the cursor position. That is, the color of the auxiliary lines is dynamically adjusted according to the pixels in different areas of the source image, making the color of the auxiliary lines different from the displayed content of the source image, which helps annotators improve annotation efficiency. The auxiliary lines are crosshairs that follow the mouse position when the user annotates the location of defects in the source image, assisting the annotator in determining the defect location.

[0197] Optionally, the line format of the guide lines can also be adjusted, including but not limited to at least one of line thickness, line style (such as solid or dashed lines), and line color. Optionally, the line format of the guide lines supports manual and / or automatic adjustment. Users can manually edit the line format of the guide lines, or automatically adjust the line format of the guide lines according to the displayed source image.

[0198] In some embodiments, the color of the guide lines is automatically adjusted in the following ways:

[0199] a) Calculate the average pixel value of the adjacent region, and invert the average pixel value to obtain the inverted pixel value;

[0200] Since the content and color of the source images vary in actual use, the crosshair guide line with a fixed color may not be obvious in some source images. Therefore, in this embodiment, the pixel image of the surrounding area of ​​the cursor position is dynamically detected, and the color of the guide line is determined by calculating the pixel mean and inverting the pixel value, so that the color of the guide line is as obviously different from the background color of the source image as possible.

[0201] In some embodiments, the adjacent regions are determined by any of the following methods:

[0202] Method a1) Determine the adjacent region based on a pixel region of size n×m centered on the target location; where n represents an integer greater than 0 and m represents an integer greater than 0;

[0203] Method a2) Determine the adjacent region based on the n×m pixel region containing the target location; where n represents an integer greater than 0 and m represents an integer greater than 0.

[0204] b) Adjust the color of the auxiliary line based on the difference between the pixel mean and the pixel inverted value.

[0205] Optionally, if the average difference between the R, G, and B channels is greater than the pixel threshold, the color of the auxiliary line is adjusted according to the inverted value of the pixel; if the average difference between the R, G, and B channels is less than or equal to the pixel threshold, the color of the auxiliary line is adjusted according to any one of red, blue, or yellow.

[0206] As shown in Figure 7, this embodiment provides a schematic diagram of the automatic adjustment principle of the auxiliary line color. First, a 5×5 pixel area around the cursor position is dynamically detected, and the average RGB pixel values ​​of these 25 pixel areas are calculated. For example, the average RGB pixel values ​​are calculated as [153, 122, 105]. The inverted results of the average RGB pixel values ​​are: [255-153, 255-122, 255-105] = [102, 133, 150]. At this time, it is necessary to analyze the difference in pixel values ​​before and after the transformation (i.e., the difference between the average pixel value and the inverted pixel value). If the average difference is greater than the pixel threshold (e.g., 25), the inverted color is directly used as the color of the cross auxiliary line. Otherwise, it means that the RGB pixel values ​​are all close to 255 / 2 = 127.5. At this time, it means that the source image tends to be a gray background, and the auxiliary line can take any one of red, blue, or yellow as the color of the cross auxiliary line. Referring to Figures 8A-8B, which provide a comparison of auxiliary lines, Figure 8A shows a fixed color. When the difference between the pixel mean and the pixel inverted value is less than or equal to the pixel threshold, the color of the auxiliary line is very close to the background color of the source image and is not easy to distinguish. Figure 8B shows the auxiliary line color adjusted to red when the difference between the pixel mean and the pixel inverted value is less than or equal to the pixel threshold. Compared with the source image with a gray background, the color of the auxiliary line is more obvious and easier to distinguish.

[0207] This embodiment also provides enhanced annotation assistance, improving annotation visibility against complex backgrounds through adjustable annotation crosshair guidelines and automatic color adjustment. The guide line color is automatically adjusted based on the image background to enhance annotation visibility.

[0208] In some embodiments, this embodiment also provides a method for category statistics, specifically including any one or more of the following statistical methods:

[0209] a) Count and display the number of labeled images in different categories;

[0210] b) Count and display the ratio of the number of labeled images in the same category to the total number of labeled images in each category.

[0211] In implementation, to facilitate real-time viewing of the number of labeled categories and the proportion of labeled images, algorithm engineers can determine algorithm optimization schemes and gain an intuitive understanding of the orientation of newly added sample images under this statistical condition. As shown in Figures 9A-9B, this embodiment provides a schematic diagram of a category statistics display interface. Based on the number of labeled images (i.e., the number of labeled boxes) and the proportion (i.e., the ratio of the number of labeled images in the same category to the total number of labeled images in all categories) statistically shown in the figure, the sample images of category NG_A can be optimized to ensure a balanced number of sample images, thereby optimizing the sample image data and thus optimizing the labeling algorithm.

[0212] This embodiment improves user experience by providing overall progress and category statistics tables, allowing users to monitor annotation progress in real time and optimize workflows. It also provides real-time statistical data to help algorithm developers refine their annotation strategies.

[0213] In some embodiments, this embodiment also provides one or more of the following filtering methods, as detailed below:

[0214] Method 1) In response to the first filter condition entered by the user, filter out labeled images of the specified category;

[0215] For example, if a user enters the name of a specified category in the filtering interface of the target application, then the labeled images of the specified category will be filtered out from the locally stored labeled images.

[0216] Method 2) In response to the second filter condition entered by the user, filter out the unlabeled source images;

[0217] For example, if a user enters unlabeled field information in the filtering interface of the target application, the unlabeled source images will be filtered out from the source images stored locally.

[0218] Method 3) In response to the third filter condition input by the user, filter out the source images without defects.

[0219] For example, if a user enters the field information indicating that the image is defect-free in the filtering interface of the target application, then defect-free source images will be filtered out from the source images stored locally.

[0220] When a detection model is trained and found to perform poorly in recognizing certain categories, and users want to quickly view the actual annotation status of that category, current annotation software requires flipping through images one by one to find the desired category. The target application provided in this embodiment offers a convenient image filtering function. The application calculates all current annotation information, summarizes the information for each category, and internally creates a mapping relationship between categories, annotated images, and source images. Each category corresponds to an array of annotated images. When switching to a specific category, the header of the current annotation interface displays the index position of the corresponding image array for that category's defects. Switching to the "previous" or "next" image automatically switches within the image array corresponding to that category. As shown in Figure 10, this embodiment provides a schematic diagram of an image filtering interface. Users can input a specified category to filter annotated images for category NG_B. The target application's display interface shows all annotated images for category NG_B.

[0221] This embodiment provides an intelligent image filtering function, allowing users to quickly filter images based on specific conditions. Through the convenient image filtering function, users can quickly locate images of specific categories or states, improving the efficiency of annotation and review.

[0222] In some embodiments, as shown in Figure 11, this embodiment also provides an image-level defect display interface, which displays labeled images corresponding to various categories of annotation data in each source image, such as labeled images for category NG_A and category NG_B. This function allows annotators to quickly view images of small defect areas (i.e., labeled images) on large-size images. In common industrial defect images, the original image usually has a high resolution, and the defects are very small. After annotating a source image, the actual defect occupies a very small proportion of the entire source image, and it is difficult to detect an abnormal annotation error. Therefore, this embodiment designs a defect display interface that allows for easy viewing of the perspective of each small defect. From this perspective, it is also easier to detect incorrect annotations of the same type.

[0223] This embodiment also enhances annotation interactivity, with newly added viewing angles and error correction functions for each annotation category, allowing annotators to perform annotation and error checking more intuitively and accurately. It allows users to quickly view and identify small-sized defect areas, improving annotation accuracy.

[0224] In some embodiments, the source image in this embodiment also includes annotation user information corresponding to each annotation data, facilitating tracing back to the source when annotation data is incorrect. As shown in Figures 12A-12B, this embodiment provides a schematic diagram of the annotation user information input function interface. When annotation users annotate the source image in the target application, an annotation user information input function is added. Referring to Figure 12A, annotation user information (such as annotation user identifier / employee number) is entered into the annotation data, making it convenient to find the relevant annotation personnel and make timely adjustments when erroneous samples are traced back. Referring to Figure 12B, for annotations where the employee number has not been logged in, the target application prompts that the annotation operation cannot be performed.

[0225] This embodiment can also improve the management of labeled data, and through the label user information entry function, enable rapid backtracking, adjustment and accountability for erroneous samples.

[0226] Based on the same inventive concept, this disclosure also provides an electronic device. Since this electronic device is the same as the electronic device in the method of this disclosure, and the principle of solving the problem by this electronic device is similar to that of this method, the implementation of this electronic device can refer to the implementation of the method, and the repeated parts will not be described again.

[0227] As shown in Figure 13, the electronic device includes a processor 1300 and a memory 1301. The memory 1301 is used to store programs executable by the processor 1300. The processor 1300 is used to read the programs in the memory 1301 and perform the following steps:

[0228] Obtain a source image library, the source image library including at least one source image and multiple annotation data of the at least one source image, the annotation data being used to represent annotation information obtained by annotating defects in the source image, the annotation data including location information;

[0229] Based on the location information of the multiple annotation data of the at least one source image, multiple annotated images are extracted from the at least one source image, wherein the annotation data and the annotated images are in one-to-one correspondence.

[0230] Display multiple labeled images corresponding to the at least one source image.

[0231] As an optional implementation, the processor 1300 is specifically configured to acquire the annotation data of the source image in the following manner:

[0232] In response to a user's annotation operation on defects in a source image within a target application, the annotation data of the source image is obtained; or,

[0233] Obtain the first detection result of the detection model on the source image for defect detection, and determine the annotation data of the source image based on the first detection result.

[0234] As an optional implementation, the processor 1300 is specifically configured to execute:

[0235] Based on the position information of each of the multiple annotation data of the source image, the source image is cropped to obtain a cropped image corresponding to the multiple annotation data;

[0236] Multiple cropped images are scaled proportionally to obtain the multiple labeled images; wherein the longest side of the scaled cropped image is within a preset range.

[0237] As an optional implementation, the processor 1300 is specifically configured to execute:

[0238] According to the category of the labeled data in the source image, display the labeled images corresponding to the labeled data; or,

[0239] Display the labeled images corresponding to the labeled data of each category in the source image, based on the different source images.

[0240] As an optional implementation, the processor 1300 is specifically configured to execute:

[0241] Based on the number and size of the labeled images, determine the target memory size required to display multiple labeled images corresponding to the at least one source image;

[0242] Based on the target memory size and the available physical memory of the target application, the multiple labeled images are saved to the specified storage location;

[0243] Based on the specified storage location, display multiple labeled images corresponding to the at least one source image.

[0244] As an optional implementation, the processor 1300 is specifically configured to determine the target memory size in the following manner:

[0245] Based on the number and size of the labeled images, determine the total runtime memory required to display the labeled images corresponding to each category of labeled data in the at least one source image;

[0246] The total running memory size is determined as the target memory size.

[0247] As an optional implementation, the processor 1300 is specifically configured to execute:

[0248] If the available physical memory meets the target memory size, then the multiple labeled images are loaded into the physical memory of the target application;

[0249] If the available physical memory does not meet the target memory size, then the multiple labeled images are saved to the local disk.

[0250] As an optional implementation, the designated storage location includes a local disk; the processor 1300 is specifically configured to execute:

[0251] According to the category of the labeled data in the source image, the labeled image corresponding to each category of labeled data is saved to the same directory on the local disk.

[0252] As an optional implementation, the storage name of the labeled image on the local disk is determined based on at least one of the following: the name of the source image, the category of the labeled data, and the location information of the labeled data.

[0253] As an optional implementation, the processor 1300 is specifically configured to perform any one or more of the following:

[0254] The specified storage location includes the physical memory of the target application, and the multiple labeled images are displayed in the first application interface of the target application;

[0255] The specified storage location includes a local disk, and the multiple labeled images are displayed in the local resource management interface;

[0256] The specified storage location includes a local disk. The multiple labeled images are displayed in the local resource management interface. In response to the user dragging and dropping the labeled images to the target application, the multiple labeled images are displayed in the second application interface of the target application.

[0257] As an optional implementation, the processor 1300 is further configured to perform:

[0258] In response to the user's viewing operation on the displayed labeled image, the location information of the labeled image is determined, and the user is redirected to the display interface of the source image corresponding to the labeled image;

[0259] Based on the location information, the source image corresponding to the labeled image and the labeling data corresponding to the labeled image are displayed on the display interface.

[0260] As an optional implementation, the processor 1300 is specifically configured to execute:

[0261] The target application is used to record the correspondence between the positional information of each labeled image and the positional information of each labeled image in the source image;

[0262] Based on the correspondence, the location information of the labeled image in the source image is determined.

[0263] As an optional implementation, the labeled image is stored on a local disk; the processor 1300 is specifically configured to execute:

[0264] The location information of the labeled image in the source image is determined based on the storage name of the labeled image in the local directory of the local disk.

[0265] As an optional implementation, the processor 1300 is further configured to perform:

[0266] Obtain the second detection result of the detection model for defect detection of the source image, and verify the accuracy of the annotation data of the source image based on the second detection result.

[0267] As an optional implementation, the processor 1300 is specifically configured to perform defect detection on the source image using a detection model in the following manner:

[0268] Obtain a detection model that meets the custom API interface standard;

[0269] The detection model is dynamically loaded based on the current hardware resources, and the loaded detection model is used to detect defects in the source image.

[0270] As an optional implementation, the processor 1300 is specifically configured to execute:

[0271] In response to a model loading request from a target application, based on current hardware resources and the number of currently loaded detection models, determine whether it is necessary to unload already loaded detection models before loading the detection model requested by the target application.

[0272] As an optional implementation, the processor 1300 is specifically configured to determine whether a loaded detection model needs to be unloaded in the following manner:

[0273] If the number of currently loaded detection models reaches the threshold, the detection model that has been requested the least among the loaded detection models will be unloaded, and the detection model requested by the target application will be loaded.

[0274] The threshold is determined based on the current hardware resources, and the number of requests indicates the number of times the detection model is requested to be loaded by the target application.

[0275] As an optional implementation, the processor 1300 is specifically configured to execute:

[0276] Determine the degree of matching between the defect image corresponding to the second detection result and the labeled image corresponding to the labeled data;

[0277] The matching degree is used to determine whether there are missing and / or incorrectly labeled annotation data in the source image.

[0278] As an optional implementation, the processor 1300 is further configured to perform:

[0279] The missing and / or incorrect labeling data are displayed in the source image; wherein the display formats of the missing and incorrect labeling data are different.

[0280] As an optional implementation, the processor 1300 is specifically configured to execute:

[0281] In response to the API interface address information entered by the user in the third application interface of the target application, the second detection result of the detection model performing defect detection on the source image is obtained.

[0282] As an optional implementation, the processor 1300 is further configured to perform:

[0283] Display auxiliary lines used for labeling;

[0284] The color of the guide line is adjusted based on the pixels in the adjacent area at the cursor position.

[0285] As an optional implementation, the processor 1300 is specifically configured to execute:

[0286] Calculate the average pixel value of the adjacent regions, and invert the average pixel value to obtain the inverted pixel value;

[0287] The color of the auxiliary line is adjusted based on the difference between the average pixel value and the inverted pixel value.

[0288] As an optional implementation, the processor 1300 is specifically configured to determine the adjacent regions in the following manner:

[0289] The adjacent region is determined based on a pixel region of size n×m centered at the target location; where n represents an integer greater than 0, and m represents an integer greater than 0; or,

[0290] The adjacent region is determined based on a pixel region of size n×m containing the target location; where n represents an integer greater than 0 and m represents an integer greater than 0.

[0291] Based on the same inventive concept, this disclosure also provides a processing apparatus for annotated images. Since this apparatus is the same as the apparatus in the method of this disclosure, and the principle of the apparatus in solving the problem is similar to that of the method, the implementation of the apparatus can refer to the implementation of the method, and the repeated parts will not be described again.

[0292] As shown in Figure 14, the device includes:

[0293] Image acquisition module 1400 is used to acquire a source image library, the source image library including at least one source image and multiple annotation data of the at least one source image, the annotation data being used to represent annotation information obtained by annotating defects in the source image, the annotation data including location information;

[0294] The image extraction module 1401 is used to extract multiple labeled images from the at least one source image based on the location information of the multiple labeled data of the at least one source image, wherein the labeled data and the labeled images are in one-to-one correspondence.

[0295] The image display module 1402 is used to display multiple labeled images corresponding to the at least one source image.

[0296] Based on the same inventive concept, this disclosure provides a computer storage medium comprising: computer program code, which, when executed on a computer, causes the computer to perform any of the annotation image processing methods described above. Since the principle by which the computer storage medium solves the problem is similar to that of the annotation image processing method, the implementation of the computer storage medium can be found in the implementation of the method, and repeated details will not be elaborated further.

[0297] In specific implementation, computer storage media can include: Universal Serial Bus Flash Drive (USB), portable hard drive, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0298] Based on the same inventive concept, this disclosure also provides a computer program product, which includes computer program code that, when executed on a computer, causes the computer to perform any of the annotation image processing methods described above. Since the principle by which the above computer program product solves the problem is similar to that of the annotation image processing method, the implementation of the above computer program product can be referred to the implementation of the method, and repeated details will not be elaborated further.

[0299] Computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0300] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0301] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0302] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device that implements the functions specified in one or more flowcharts and / or one or more block diagrams.

[0303] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0304] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for processing labeled images, wherein, The method includes: Obtain a source image library, the source image library including at least one source image and multiple annotation data of the at least one source image, the annotation data being used to represent annotation information obtained by annotating defects in the source image, the annotation data including location information; Based on the location information of the multiple annotation data of the at least one source image, multiple annotated images are extracted from the at least one source image, wherein the annotation data and the annotated images are in one-to-one correspondence. Display multiple labeled images corresponding to the at least one source image.

2. The method according to claim 1, wherein, The annotation data of the source image is obtained in the following manner: In response to a user's annotation operation on defects in a source image within a target application, the annotation data of the source image is obtained; or, Obtain the first detection result of the detection model on the source image for defect detection, and determine the annotation data of the source image based on the first detection result.

3. The method according to claim 1, wherein, The step of extracting multiple labeled images from the at least one source image based on the location information of each of the multiple labeled data of the at least one source image includes: Based on the position information of each of the multiple annotation data of the source image, the source image is cropped to obtain a cropped image corresponding to the multiple annotation data; Multiple cropped images are scaled proportionally to obtain the multiple labeled images; wherein the longest side of the scaled cropped image is within a preset range.

4. The method according to claim 1, wherein, The display of multiple labeled images corresponding to the at least one source image includes: According to the category of the labeled data in the source image, display the labeled images corresponding to the labeled data; or, Display the labeled images corresponding to the labeled data of each category in the source image, based on the different source images.

5. The method according to claim 1, wherein, The display of multiple labeled images corresponding to the at least one source image includes: Based on the number and size of the labeled images, determine the target memory size required to display multiple labeled images corresponding to the at least one source image; Based on the target memory size and the available physical memory of the target application, the multiple labeled images are saved to the specified storage location; Based on the specified storage location, display multiple labeled images corresponding to the at least one source image.

6. The method according to claim 5, wherein, The target memory size is determined as follows: Based on the number and size of the labeled images, determine the total runtime memory required to display the labeled images corresponding to each category of labeled data in the at least one source image; The total running memory size is determined as the target memory size.

7. The method according to claim 5, wherein, Based on the target memory size and the available physical memory of the target application, the multiple labeled images are saved to a specified storage location, including: If the available physical memory meets the target memory size, then the multiple labeled images are loaded into the physical memory of the target application; If the available physical memory does not meet the target memory size, then the multiple labeled images are saved to the local disk.

8. The method according to claim 5, wherein, The specified storage location includes a local disk; saving the plurality of labeled images to the specified storage location includes: According to the category of the labeled data in the source image, the labeled image corresponding to each category of labeled data is saved to the same directory on the local disk.

9. The method according to claim 8, wherein, The storage name of the labeled image on the local disk is determined based on at least one of the following: the name of the source image, the category of the labeled data, and the location information of the labeled data.

10. The method according to claim 5, wherein, The step of displaying multiple labeled images corresponding to the at least one source image based on the specified storage location includes any one or more of the following: The specified storage location includes the physical memory of the target application, and the multiple labeled images are displayed in the first application interface of the target application; The specified storage location includes a local disk, and the multiple labeled images are displayed in the local resource management interface; The specified storage location includes a local disk. The multiple labeled images are displayed in the local resource management interface. In response to the user dragging and dropping the labeled images to the target application, the multiple labeled images are displayed in the second application interface of the target application.

11. The method according to any one of claims 1 to 10, wherein, The method also includes: In response to the user's viewing operation on the displayed labeled image, the location information of the labeled image is determined, and the user is redirected to the display interface of the source image corresponding to the labeled image; Based on the location information, the source image corresponding to the labeled image and the labeling data corresponding to the labeled image are displayed on the display interface.

12. The method according to claim 11, wherein, Determining the location information of the labeled image includes: The target application is used to record the correspondence between the positional information of each labeled image and the positional information of each labeled image in the source image; Based on the correspondence, the location information of the labeled image in the source image is determined.

13. The method according to claim 11, wherein, The labeled image is stored on a local disk; determining the location information of the labeled image includes: The location information of the labeled image in the source image is determined based on the storage name of the labeled image in the local directory of the local disk.

14. The method according to any one of claims 1 to 10, 12 to 13, wherein, The method also includes: Obtain the second detection result of the detection model for defect detection of the source image, and verify the accuracy of the annotation data of the source image based on the second detection result.

15. The method according to claim 14, wherein, Defect detection in the source image is performed using the detection model in the following manner: Obtain a detection model that meets the custom API interface standard; The detection model is dynamically loaded based on the current hardware resources, and the loaded detection model is used to detect defects in the source image.

16. The method according to claim 15, wherein, The dynamic loading of the detection model based on current hardware resources includes: In response to a model loading request from a target application, based on current hardware resources and the number of currently loaded detection models, determine whether it is necessary to unload already loaded detection models before loading the detection model requested by the target application.

17. The method according to claim 16, wherein, Determine whether the loaded detection model needs to be uninstalled using the following method: If the number of currently loaded detection models reaches the threshold, the detection model that has been requested the least among the loaded detection models will be unloaded, and the detection model requested by the target application will be loaded. The threshold is determined based on the current hardware resources, and the number of requests indicates the number of times the detection model is requested to be loaded by the target application.

18. The method according to claim 14, wherein, The verification of the accuracy of the annotation data of the source image based on the second detection result includes: Determine the degree of matching between the defect image corresponding to the second detection result and the labeled image corresponding to the labeled data; The matching degree is used to determine whether there are missing and / or incorrectly labeled annotation data in the source image.

19. The method according to claim 18, wherein, The method also includes: The missing and / or incorrect labeling data are displayed in the source image; wherein the display formats of the missing and incorrect labeling data are different.

20. The method of claim 14, wherein, The second detection result obtained by the detection model for defect detection of the source image includes: In response to the API interface address information entered by the user in the third application interface of the target application, the second detection result of the detection model performing defect detection on the source image is obtained.

21. The method according to any one of claims 1 to 10, 12 to 13, and 15 to 20, wherein, The method also includes: Display auxiliary lines used for labeling; The color of the guide line is adjusted based on the pixels in the adjacent area at the cursor position.

22. The method according to claim 21, wherein, Adjust the color of the auxiliary line based on the pixels in the adjacent area at the cursor position, including: Calculate the average pixel value of the adjacent regions, and invert the average pixel value to obtain the inverted pixel value; The color of the auxiliary line is adjusted based on the difference between the average pixel value and the inverted pixel value.

23. The method according to claim 21, wherein, The adjacent regions are determined in the following manner: The adjacent region is determined based on a pixel region of size n×m centered at the target location; where n represents an integer greater than 0, and m represents an integer greater than 0; or, The adjacent region is determined based on a pixel region of size n×m containing the target location; where n represents an integer greater than 0 and m represents an integer greater than 0.

24. An electronic device, wherein, The electronic device includes a processor and a memory, the memory being used to store a program executable by the processor, and the processor being used to read the program in the memory and execute the steps of the method according to any one of claims 1 to 23.

25. A computer storage medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 23.

Images