Target marker annotation method and apparatus, device, and storage medium

By constructing 3D point cloud data and combining it with cloud-based models and user interaction, semi-automatic annotation of traffic signs was achieved, solving the problem of high cost in existing methods and improving annotation efficiency and quality.

WO2026237891A1PCT designated stage Publication Date: 2026-11-19BEIJING HORIZON INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/096195
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-16
Filing Date
2026-05-11
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing methods for marking traffic signs are mainly based on images, resulting in high marking costs.

Method used

By determining the time-series data of the target scene, constructing 3D point cloud data, using a large cloud-based annotation model to obtain predicted locations and initial annotation results, and displaying target information on the user interface, the system responds to user operations to perform semi-automatic annotation adjustments, thereby achieving rapid annotation of target markers.

Benefits of technology

While ensuring annotation quality, it improved annotation efficiency and reduced annotation costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026096195_19112026_PF_FP_ABST
    Figure CN2026096195_19112026_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed are a target marker annotation method and apparatus, a device, and a storage medium. The method comprises: constructing three-dimensional point cloud data on the basis of time-series data of a target scene, and determining a predicted position of a target marker in the three-dimensional point cloud data; acquiring a first annotation result of the target marker in the three-dimensional point cloud data; on the basis of the predicted position and the first annotation result, determining target information of the target marker, and displaying the target information on a user interface; and in response to an annotation operation performed on the target information on the user interface, determining a second annotation result of the target marker in the three-dimensional point cloud data. In the technical solution of the present disclosure, pre-annotation and prediction are performed for the target marker by means of the three-dimensional point cloud of the target scene to obtain a pre-annotation result and a prediction result; subsequently, the pre-annotation result and the prediction result are used to assist manual annotation of the target marker. In this way, rapid annotation of the target marker can be achieved on the basis of a semi-automated human-computer interaction approach, thereby improving annotation efficiency while ensuring annotation quality.
Need to check novelty before this filing date? Find Prior Art

Description

Methods, apparatus, equipment and storage media for labeling target markers

[0001] This disclosure claims priority to Chinese Patent Application No. 2025106392334, filed on May 16, 2025, entitled “Method and Apparatus, Device and Storage Medium for Marking Target Markers”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of data annotation, and in particular to a method, apparatus, device and storage medium for annotating target markers. Background Technology

[0003] Model training is an essential part of realizing autonomous driving, and the training process typically requires a large amount of labeled data, making data labeling a crucial step. It is well known that the higher the accuracy of the data labeling, the better the performance of the trained algorithm model.

[0004] In intelligent driving perception, the accuracy of perceiving traffic signs (such as traffic lights and signs) is crucial for the safety of advanced driver assistance systems (ADAS). Training perception models requires a large number of accurate labeled samples. However, most current traffic sign labeling methods in the industry are based on images, resulting in high labeling costs. Summary of the Invention

[0005] Currently, most methods for labeling traffic signs in intelligent driving scenarios are based on images, which requires a large number of labeled samples, resulting in very high labeling costs.

[0006] To address the aforementioned technical problems, this disclosure provides a method, apparatus, device, and storage medium for marking target markers, which can solve the problem of high marking costs in existing traffic sign marking methods.

[0007] The first aspect of this disclosure provides a method for labeling target markers, comprising: determining time-series data corresponding to a target scene; constructing three-dimensional point cloud data of the target scene based on the time-series data, and determining the predicted position of the target marker in the three-dimensional point cloud data; obtaining a first labeling result of the target marker in the three-dimensional point cloud data; determining target information corresponding to the target marker based on the predicted position and the first labeling result, and displaying the target information on a user interface; and determining a second labeling result of the target marker in the three-dimensional point cloud data in response to detecting a labeling operation on the target information on the user interface.

[0008] A second aspect of this disclosure provides a target marker annotation device, comprising: a time-series data determination module for determining time-series data corresponding to a target scene; a prediction position determination module for constructing three-dimensional point cloud data of the target scene based on the time-series data and determining the predicted position of the target marker in the three-dimensional point cloud data; a pre-brush result determination module for acquiring a first annotation result of the target marker in the three-dimensional point cloud data; an annotation display module for determining target information corresponding to the target marker based on the predicted position and the first annotation result, and displaying the target information on a user interface; and an annotation result determination module for determining a second annotation result of the target marker in the three-dimensional point cloud data in response to detecting an annotation operation on the target information on the user interface.

[0009] A third aspect of this disclosure provides a computer-readable storage medium storing a computer program for performing the annotation method for target markers provided in the first aspect above.

[0010] A fourth aspect of this disclosure provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to read the executable instructions from the memory and execute the instructions to: determine time-series data corresponding to a target scene; construct three-dimensional point cloud data of the target scene based on the time-series data and determine the predicted position of a target marker in the three-dimensional point cloud data; acquire a first annotation result of the target marker in the three-dimensional point cloud data; determine target information corresponding to the target marker based on the predicted position and the first annotation result, and display the target information on a user interface; and, in response to detecting an annotation operation on the target information on the user interface, determine a second annotation result of the target marker in the three-dimensional point cloud data.

[0011] A fifth aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, performs the labeling method for the target markers provided in the first aspect above.

[0012] Based on the target marker annotation method provided in this disclosure, the following steps are taken: First, determine the temporal data corresponding to the target scene; second, construct 3D point cloud data of the target scene based on the temporal data and determine the predicted position of the target marker in the 3D point cloud data; third, obtain the first annotation result of the target marker in the 3D point cloud data; fourth, determine the target information corresponding to the target marker based on the predicted position and the first annotation result, and display the target information on the user interface; fifth, in response to detecting the annotation operation on the target information on the user interface, determine the second annotation result of the target marker in the 3D point cloud data. In this way, the 3D point cloud data constructed from the temporal data and the pre-printed annotation result of the target marker (the target information corresponding to the target marker) can be displayed on the user interface, allowing for manual annotation adjustment operations. This semi-automatic human-computer interaction method enables rapid annotation of target markers, improving annotation efficiency while ensuring the quality of target marker annotation. Attached Figure Description

[0013] Figure 1 is a schematic diagram of an application scenario provided by an exemplary embodiment of this disclosure.

[0014] Figure 2A is a flowchart illustrating an annotation method provided in an exemplary embodiment of this disclosure.

[0015] Figure 2B is a schematic diagram of a user interface provided in an exemplary embodiment of this disclosure.

[0016] Figure 3 is a flowchart illustrating a labeling method provided in another exemplary embodiment of this disclosure.

[0017] Figure 4A is a flowchart illustrating a labeling method provided in yet another exemplary embodiment of this disclosure.

[0018] Figure 4B is a schematic diagram of three views provided in yet another exemplary embodiment of this disclosure.

[0019] Figure 4C is a schematic diagram of a user interface provided by another exemplary embodiment of this disclosure.

[0020] Figure 5A is a flowchart illustrating a labeling method provided in yet another exemplary embodiment of this disclosure.

[0021] Figure 5B is a schematic diagram of a user interface provided in yet another exemplary embodiment of this disclosure.

[0022] Figure 5C is a schematic diagram of an image projection result provided by an exemplary embodiment of the present disclosure.

[0023] Figure 6 is a flowchart illustrating the annotation method provided in yet another exemplary embodiment of this disclosure.

[0024] Figure 7 is a flowchart illustrating a labeling method provided in yet another exemplary embodiment of this disclosure.

[0025] Figure 8 is a schematic diagram of the structure of a labeling device provided in an exemplary embodiment of this disclosure.

[0026] Figure 9 is a schematic diagram of the structure of a labeling device provided in another exemplary embodiment of this disclosure.

[0027] Figure 10 is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of the present disclosure. Detailed Implementation

[0028] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0029] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0030] Application Overview

[0031] The target marker annotation method provided in this disclosure can be applied to scenarios involving ground truth annotation during model training, as well as any other feasible scenarios. For example, it can be applied to training samples required for perceptual model training: when training a perceptual model for detecting target markers (such as traffic signs, traffic lights, etc.), the method can first use this disclosure to annotate multiple image sequences to be labeled using image sensors (also known as vision sensors) with different orientations to obtain annotation results. These annotation results can then be used as training samples to train the perceptual model until the preset training completion conditions are met, resulting in a trained perceptual model that can be used for downstream perceptual tasks.

[0032] Figure 1 is a schematic diagram of an application scenario provided by an exemplary embodiment of this disclosure. As shown in Figure 1, a vehicle 10 is equipped with multiple image sensors 11 (e.g., cameras) and a 3D scanning device 12 (e.g., LiDAR) facing different directions. The vehicle 10 passes through a target scene 13 at a preset driving speed. During the driving of the vehicle 10, each image sensor 11 on the vehicle 10 acquires images of the target scene 13 at a first preset frequency, obtaining a sequence of images to be labeled. Therefore, multiple image sensors 11 correspond to multiple sequences of images to be labeled; wherein, each sequence of images to be labeled includes multiple frames of images to be labeled, and the multiple frames of images to be labeled in each sequence are sorted according to the image acquisition time or the driving trajectory of the vehicle 10. During the driving of the vehicle 10, the 3D scanning device 12 can also acquire point cloud data of the target scene 13 at a second preset frequency, obtaining a point cloud dataset. The first preset frequency and the second preset frequency can be the same or different. Based on the above multiple sequences of images to be labeled and / or point cloud datasets, a 3D scene reconstruction is performed on the target scene 13 to obtain the 3D point cloud data of the target scene 13. Furthermore, the target marker annotation method provided in this embodiment is used to annotate the three-dimensional point cloud data to obtain the annotation results of the target markers (such as traffic signs, traffic lights, buildings, etc.) in the three-dimensional point cloud. Further, the annotation results of the target markers in the three-dimensional point cloud data can be projected onto the aforementioned multiple image sequences to be annotated, resulting in multiple annotated image sequences with the annotation results of the target markers.

[0033] It should be noted that Figure 1 is merely an illustrative example, and the present disclosure does not limit the number and position of the multiple image sensors 11 and the 3D scanning device 12 in Figure 1. In actual use, the image sensor 11 may be the image sensor corresponding to the vehicle's forward-looking wide-angle camera, the image sensor 11 may be the image sensor corresponding to the vehicle's forward-looking narrow-angle camera, or the image sensor 11 may be the image sensor corresponding to a side-view camera, etc.

[0034] Specifically, this disclosure provides a method for labeling target markers. This method involves determining time-series data corresponding to a target scene; constructing 3D point cloud data of the target scene based on the time-series data; determining the predicted position of the target marker in the 3D point cloud data; obtaining a first labeling result for the target marker in the 3D point cloud data; determining the target information corresponding to the target marker based on the predicted position and the first labeling result; and displaying the target information on a user interface. In response to detecting a labeling operation on the target information on the user interface, a second labeling result for the target marker in the 3D point cloud data is determined. This allows the display of the 3D point cloud data constructed from the time-series data and the pre-labeled result of the target marker (the target information corresponding to the target marker) on the user interface, enabling manual labeling adjustments. This semi-automatic human-computer interaction method facilitates rapid labeling of target markers, improving labeling efficiency while ensuring labeling quality.

[0035] Exemplary methods

[0036] Figure 2A is a flowchart illustrating a method for labeling target markers according to an embodiment of this disclosure. This embodiment can be applied to any electronic device such as a local terminal device or a cloud server, or it can be distributed across terminal devices and cloud servers. As shown in Figure 2A, the method includes the following steps S201-S205.

[0037] Step S201: Determine the time series data corresponding to the target scene.

[0038] For example, the target scenario may include a scene on a section of a highway, or a scene at an intersection on a city road, etc. The actual situation of the target scenario is not limited in this embodiment of the disclosure. The factors affecting the target scenario include, but are not limited to: driving location, road type, weather conditions, or traffic conditions.

[0039] In some examples, as shown in Figure 1, vehicle 10 passes through target scene 13 at a preset speed. Each image sensor 11 on vehicle 10 acquires images of target scene 13 at a first preset frequency, obtaining multiple image sequences to be labeled. These multiple image sequences are the time-series data in step S201. The image acquisition frequencies of each image sensor 11 can be the same or different; this embodiment does not impose any limitations on this. Similarly, as shown in Figure 1, vehicle 10 passes through target scene 13 at a preset speed. The 3D scanning device 12 on vehicle 10 scans target scene 13 at a second preset frequency, obtaining multiple point cloud data with a temporal relationship. These multiple point cloud data are the time-series data in step S201. In other words, the time-series data corresponding to the target scene in this embodiment includes, but is not limited to, image sequences acquired at the first preset frequency and / or point cloud datasets acquired at the second preset frequency in the target scene.

[0040] Step S202: Construct three-dimensional point cloud data of the target scene based on time-series data, and determine the predicted position of the target marker in the three-dimensional point cloud data.

[0041] For example, the temporal data corresponding to the target scene can be multiple image sequences to be labeled, and / or multiple point cloud data with temporal relationships. Based on this, the target scene can be reconstructed using the multiple point cloud data with temporal relationships to obtain the 3D point cloud data of the target scene. The 3D scanning device includes, but is not limited to, LiDAR (Light Laser Detection and Ranging), structured light sensors, or TOF (Time of Flight) cameras. Alternatively, the target scene can also be reconstructed using the image sequences to be labeled to obtain the 3D point cloud data of the target scene. Of course, multiple point cloud data with temporal relationships and image sequences to be labeled can be used simultaneously for 3D reconstruction to improve the accuracy of the 3D reconstruction results. This disclosure does not limit the specific implementation method for constructing the 3D point cloud data of the target scene.

[0042] In some examples, the target scene of this disclosure embodiment may contain one or more target landmarks (such as traffic lights or traffic signs). For example, multiple traffic lights at an intersection in a city; or signs on a section of a highway. Furthermore, the predicted position of the target landmark in the 3D point cloud data can be determined based on the 3D point cloud data of the target scene. This predicted position indicates the presence of a target landmark at that location in the 3D point cloud data. The predicted position can be represented as a prediction bounding box, and therefore can include the 3D coordinate information of each point in the prediction bounding box within the 3D point cloud data. It should be noted that this is merely an example, and the specific representation of the predicted position of the target landmark in the 3D point cloud data in this disclosure embodiment is not limited.

[0043] Step S203: Obtain the first annotation result of the target marker in the 3D point cloud data.

[0044] For example, the first annotation result of a target marker in 3D point cloud data can be the first 3D (three-dimensional) bounding box of the target marker in the 3D point cloud data. For instance, a large cloud-based annotation model can be pre-trained, with the input of 3D point cloud data and the output of the first 3D bounding box of the target marker in the 3D point cloud data. In this way, the pre-printed result (first 3D bounding box) of the target marker can be automatically obtained using the pre-trained cloud-based annotation model, thereby eliminating the need for manual annotation and saving annotation costs and time.

[0045] It should be noted that this disclosure does not impose any restrictions on the model structure, training samples, training iteration termination conditions, or detection accuracy of the cloud-annotated large model. Those skilled in the art can set these parameters according to actual conditions during practical use. This disclosure uses the cloud-annotated large model to process 3D point cloud data to obtain corresponding pre-brush results. These pre-brush results refer to the initial annotation results of target markers in the 3D point cloud data. Of course, other methods can also be used to obtain the pre-brush results of target markers in 3D point cloud data, and this disclosure does not limit this. The cloud-annotated large model is a model deployed in the cloud that achieves automated data annotation based on a large model. A large model refers to a model with a large number of parameters, which acquires general understanding capabilities through pre-training and can automatically complete data annotation / pre-annotation / data augmentation tasks, including but not limited to: text large models, visual large models, multimodal large models, and speech large models.

[0046] Step S204: Based on the predicted location and the first annotation result, determine the target information corresponding to the target marker and display the target information on the user interface.

[0047] For example, the first annotation result can be obtained from a cloud-based annotation model. Those skilled in the art know that models typically do not achieve 100% accuracy due to various reasons (such as insufficient or overly complex training data, or structural defects in the model). Therefore, the output of the cloud-based annotation model may also contain certain errors. For instance, a target marker exists at location A in the 3D point cloud data, but the first 3D bounding box output by the cloud-based annotation model is located at location B, where location A and location B are different locations in the 3D point cloud data. Or, multiple target markers exist in the 3D point cloud data, but the cloud-based annotation model only outputs the first 3D bounding box corresponding to one target marker, failing to detect the others. Therefore, in this embodiment, the predicted location is used to verify the first annotation result, obtaining a corresponding verification result; thereby, based on the verification result, the target information corresponding to the target marker is determined, and this target information is displayed on the user interface. This target information is related to the predicted location, or it is related to the first annotation result.

[0048] In some examples, the 3D point cloud data corresponding to the target scene and the determined target information can be visualized so that the 3D point cloud data of the target scene and the determined target information can be displayed on a visual interactive interface. For example, the visualization module in PCL (Point Cloud Library) and Cloud Compare (point cloud registration) visualization software can be used to visualize the 3D point cloud data of the target scene and the determined target information. Furthermore, the aforementioned visual interactive interface (i.e., the user interface in step S204) is used not only to display the 3D point cloud data of the target scene and the determined target information, but also to edit or set the target information displayed in the 3D point cloud data according to the user's operation instructions (such as annotation instructions or adjustment instructions). The visual interactive interface can display data through a display (e.g., a liquid crystal display or a plasma display), and the visual interactive interface can also receive user operation instructions through a keyboard, mouse, or touch screen.

[0049] In some examples, Figure 2B is an exemplary schematic diagram of target information in three-dimensional point cloud data according to an embodiment of this disclosure. As shown in Figure 2B, the target marker 21 is a traffic light, and the target information 22 in the three-dimensional point cloud data can be displayed on the visualization interface 23. The target information 22 in Figure 2B is the first 3D bounding box of the target marker 21. For example, the first 3D bounding box can be a bounding box used to surround the traffic light. It should be noted that while Figure 2B contains multiple target markers and their corresponding target information, only one target marker and its corresponding target information are shown in Figure 2B for illustrative purposes.

[0050] Step S205: In response to detecting the annotation operation of target information on the user interface, determine the second annotation result of the target marker in the 3D point cloud data.

[0051] For example, after the 3D point cloud data of the target scene and the determined target information are displayed on the visual interactive interface, the user can use the visual interactive interface to annotate the target information in the 3D point cloud data, thereby determining the second annotation result of the target marker in the 3D point cloud data. The second annotation result is obtained by the user annotating the target information, and therefore is usually more accurate than the first annotation result.

[0052] The target marker annotation method provided in this embodiment determines the time-series data corresponding to the target scene; constructs 3D point cloud data of the target scene based on the time-series data, and determines the predicted position of the target marker in the 3D point cloud data; obtains the first annotation result of the target marker in the 3D point cloud data; determines the target information corresponding to the target marker based on the predicted position and the first annotation result, and displays the target information on the user interface; and determines the second annotation result of the target marker in the 3D point cloud data in response to detecting the annotation operation of the target information on the user interface. In this way, the 3D point cloud data constructed from the time-series data and the pre-printed annotation result of the target marker (the target information corresponding to the target marker) can be displayed on the user interface, allowing manual annotation adjustment operations. This achieves rapid annotation of the target marker based on a semi-automatic human-computer interaction method, improving annotation efficiency while ensuring the annotation quality of the target marker.

[0053] As shown in Figure 3, based on the embodiment shown in Figure 2A above, step S204 may include the following steps S2041-S2042.

[0054] Step S2041: Match the prediction box of the predicted location with the annotation box of the first annotation result to obtain the matching result.

[0055] For example, the predicted bounding box at the predicted location can be matched (compared) with the bounding box of the first annotation result (such as the first 3D bounding box) to obtain a matching result. The matching result includes whether the position of the bounding box in the 3D point cloud data is the same as the position of the predicted bounding box in the 3D point cloud data, or whether the position of the bounding box in the 3D point cloud data is different from the position of the predicted bounding box. For example, if the matching result indicates that a bounding box exists at the location of the predicted bounding box, the matching result indicates that the first annotation result is usable. Conversely, if the matching result indicates that a bounding box does not exist at the location of the predicted bounding box, the matching result indicates that the first annotation result is unusable.

[0056] Step S2042: Based on the matching results, determine the target information corresponding to the target marker and display the target information on the user interface.

[0057] For example, in this embodiment of the disclosure, target information corresponding to the target marker can be determined based on different matching results. For instance, if the matching result shows that the position of the bounding box in the 3D point cloud data is the same as the position of the predicted bounding box in the 3D point cloud data, then the target information is related to the first annotation result; that is, the target information can be the bounding box in the first annotation result. As another example, if the position of the bounding box in the 3D point cloud data is different from the position of the predicted bounding box in the 3D point cloud data, then the target information is related to the predicted position; that is, the target information can be the predicted bounding box at the predicted position.

[0058] The target marker annotation method provided in this embodiment of the present disclosure obtains a matching result by matching the prediction box of the predicted position with the annotation box of the first annotation result; based on the matching result, the target information corresponding to the target marker is determined; thus, the first annotation result of automatic annotation can be verified, and the annotation content to be adjusted can be determined based on the verification result, thereby improving the accuracy of annotation.

[0059] In some embodiments, determining the target information corresponding to the target marker based on the matching result includes: if the matching result shows that the predicted box and the labeled box are in the same position, determining the labeled box as the target information corresponding to the target marker; or, if the matching result shows that the predicted box and the labeled box are not in the same position, determining the predicted box as the target information corresponding to the target marker.

[0060] For example, if the matching result shows that the position of the bounding box in the 3D point cloud data is the same as the position of the predicted bounding box in the 3D point cloud data, then the target information is the bounding box of the first annotation result. Alternatively, if the position of the bounding box in the 3D point cloud data is different from the position of the predicted bounding box, then the target information is the predicted bounding box of the predicted position. Furthermore, if the target information is the bounding box of the first annotation result, then the 3D point cloud data of the target scene and the bounding box are displayed in the user interface; if the target information is the predicted bounding box of the predicted position, then the 3D point cloud data of the target scene and the predicted bounding box are displayed in the user interface. For example, if the matching result shows that a first 3D bounding box exists at the location of the predicted bounding box, then the 3D point cloud data and the first 3D bounding box are displayed in the visual interactive interface. Similarly, if the matching result shows that a first 3D bounding box does not exist at the location of the predicted bounding box, then the 3D point cloud data and the predicted bounding box are displayed in the visual interactive interface. In this way, corresponding target information can be displayed based on the automatic pre-brushing result or the prediction result for manual annotation, thereby achieving convenient, fast, and accurate annotation of target markers.

[0061] As shown in Figure 4A, based on the embodiment shown in Figure 2A above, step S205 may include the following steps S2051a-S2053a.

[0062] Step S2051a: When the target information is the annotation box of the first annotation result, in response to detecting the adjustment operation of the first sub-annotation box in the annotation box on the user interface, determine the adjusted first sub-annotation box.

[0063] For example, if the target information is the annotation box of the first annotation result, the 3D point cloud data and the annotation box are displayed on a visual interactive interface (user interface), allowing the user to adjust the first sub-annotation box of the annotation box in the 3D point cloud data using the visual interactive interface to obtain the adjusted first sub-annotation box. The annotation box of the first annotation result includes a first sub-annotation box and a second sub-annotation box, which are used to annotate different content in the target marker.

[0064] In some examples, when the time-series data includes a sequence of images to be labeled, multiple frames of images to be labeled within that sequence can be identified. Then, the bounding boxes of the first labeling result (including first and second sub-bounding boxes) can be projected onto each frame of the image to be labeled, allowing the user to verify the accuracy of the first labeling result. If the verification result indicates that the geometric and attribute information of the first and second sub-bounding boxes are correctly labeled, then in response to detecting a first click operation on the user interface, the first labeling result is determined as the second labeling result. Geometric information includes, but is not limited to, position or size (including length, width, or height). Attribute information includes, but is not limited to, orientation, shape, or type. The first click operation triggers the event of determining the first labeling result as the second labeling result.

[0065] In some examples, if the verification results indicate that the geometric and / or attribute information of the first sub-label box is incorrect, the adjusted first sub-label box is determined in response to the detection of an adjustment operation on the user interface.

[0066] For example, the target sign may include a traffic light, which includes a lightbox and a bulb. Therefore, the first sub-label box is the lightbox label box, and the second sub-label box is the bulb label box. Furthermore, if the lightbox type labeled in the lightbox label box is correct, in response to detecting adjustments to the orientation, position, and size of the lightbox label box on the user interface, an adjusted lightbox label box is obtained. If the lightbox type labeled in the lightbox label box is incorrect, in response to detecting adjustments to the lightbox type on the user interface, a lightbox label box with the adjusted type is obtained; furthermore, in response to detecting adjustments to the orientation, position, and size of the lightbox label box on the user interface, an adjusted lightbox label box is obtained.

[0067] In some examples, the first sub-annotation box can be adjusted in three views (top view, front view, and side view), which allows the first sub-annotation box to be displayed from various angles, making it convenient for users to adjust.

[0068] Step S2052a: Generate a second sub-label box based on the adjusted first sub-label box.

[0069] For example, if the verification result indicates that the geometric information and / or attribute information of the first sub-label box is incorrect, then in response to detecting an adjustment operation on the first sub-label box on the user interface, the adjusted first sub-label box is determined. Typically, the first and second sub-label boxes are related; for example, the second sub-label box may be located within the first sub-label box. Therefore, if the geometric information and / or attribute information of the first sub-label box is incorrect, the second sub-label box may also be incorrectly labeled. Furthermore, in response to detecting a deletion operation on the second sub-label box in the first annotation result on the user interface, a new second sub-label box can be generated based on the adjusted first sub-label box. Alternatively, the second sub-label box can be generated based on the adjusted first sub-label box in response to detecting a second click operation on the user interface. The second click operation triggers the generation of the second sub-label box. It should be noted that the triggering conditions for generating the second sub-label box based on the adjusted first sub-label box are not limited in this embodiment; the above two are merely illustrative examples.

[0070] In some examples, the target marker can be a traffic light, which includes a lightbox and a bulb. Therefore, the first sub-annotation box is the lightbox annotation box, and the second sub-annotation box is the bulb annotation box, located within the lightbox annotation box. Furthermore, after adjusting the lightbox annotation box, the bulb annotation box can be generated based on the adjusted lightbox annotation box. For example, a 3D bulb annotation box can be generated according to national traffic light standards based on the adjusted 3D lightbox annotation box. For instance, shrinking the lightbox by a certain proportion and then dividing it equally can generate 3D bulb annotation boxes with the same orientation. In this way, accurate and consistent ground truth data for parameters such as position and orientation can be obtained, improving annotation efficiency while ensuring annotation quality.

[0071] In some examples, after generating the second sub-annotation box, it can be fine-tuned using three views (top view, front view, and side view), and the fine-tuned second sub-annotation box can be projected onto the image domain to check the annotation quality. As shown in Figure 4B, image 41 shows the light bulb in the 3D point cloud data in the top view, image 42 shows the light bulb in the 3D point cloud data in the side view, and image 43 shows the light bulb in the 3D point cloud data in the front view; the boxes in each image are used to indicate the position of the light bulb in the 3D point cloud data, and the arrows are used to indicate the orientation of the light bulb. Furthermore, the position and size of the light bulb can be fine-tuned in the above three views, and during the fine-tuning process, it can be projected onto the image domain to check the annotation quality, finally obtaining the fine-tuned light bulb annotation box. As shown in Figure 4C, 3D annotation box 44 is the final obtained light bulb annotation box, and 3D annotation box 45 is the adjusted lightbox annotation box.

[0072] Step S2053a: Based on the second sub-label box and the adjusted first sub-label box, determine the second labeling result of the target marker in the 3D point cloud data.

[0073] For example, the adjusted first sub-label box and the generated second sub-label box can be used to determine the second labeling result of the target marker in the 3D point cloud data. For instance, as shown in Figure 4C, the light bulb 3D label box 44 and the light box 3D label box 45 together constitute the second 3D label box of the traffic signal light.

[0074] The target marker annotation method provided in this embodiment of the present disclosure, when the target information is the annotation box of the first annotation result, responds to the detection of the adjustment operation of the first sub-annotation box in the annotation box on the user interface, and determines the adjusted first sub-annotation box; generates a second sub-annotation box based on the adjusted first sub-annotation box; and determines the second annotation result of the target marker in the three-dimensional point cloud data based on the second sub-annotation box and the adjusted first sub-annotation box. In this way, the second sub-annotation box can be generated based on the adjusted first sub-annotation box, thereby conveniently, quickly and accurately realizing the annotation of the target marker.

[0075] As shown in Figure 5A, based on the embodiment shown in Figure 2A above, step S205 may include the following steps S2051b-S2052b.

[0076] S2051b When the target information is a prediction box of the predicted location, in response to detecting a selection operation on the user interface, the position information of the prediction box and the type of the target marker corresponding to the selection operation are determined.

[0077] For example, if the target information is a predicted bounding box of the predicted location, the 3D point cloud data and the predicted bounding box are displayed on a visual interactive interface (user interface), so that the user can manually annotate the target marker using the position information of the predicted bounding box on the visual interactive interface to obtain the corresponding second annotation result.

[0078] In some examples, the selection operation in step S2051b is used to select the type of the target sign. For example, the type of the target sign includes, but is not limited to, traffic lights or traffic signs. In this embodiment of the disclosure, the type of the target sign may further include the types of different contents within the target sign. For example, when the target sign is a traffic light, the traffic light includes a light box and a light bulb. Furthermore, the type of light box may also be included. As shown in Figure 5B, the operator can select whether the prediction box is marked as a traffic light or a sign, and the type of the light box of the traffic light, from the labeling directory in the user interface 51. As can be seen from Figure 5B, the light box types of traffic lights include, but are not limited to, three-way traffic lights, three-way traffic lights, two-way traffic lights, two-way traffic lights, single traffic lights, or other lights.

[0079] S2052b: Based on the location information of the predicted bounding box and the type of the target marker, generate the second annotation result of the target marker in the 3D point cloud data.

[0080] For example, based on the location information of the predicted bounding box in the 3D point cloud data and the type of the selected target marker, a second annotation result of the target marker in the 3D point cloud data can be generated at the location indicated by the location information. For example, as shown in Figure 5B, when the operator selects a traffic light as the target marker and the light box type as a horizontal double signal light in the user interface 51, a 3D annotation box corresponding to the horizontal double signal light can be generated at the location indicated by the location information in response to the operator's selection operation. When the operator confirms that the 3D annotation box of the light box is accurate, in response to the operator's second click operation, the light box can be shrunk inward by a certain proportion and then divided into two equal parts to generate a 3D annotation box of the bulb with the same orientation as the 3D annotation box of the light box. That is to say, in this embodiment of the present disclosure, by selecting the type of light box and pasting the corresponding type of light box 3D annotation box onto the location in the 3D point cloud data where the traffic light is predicted, a 3D annotation box of the bulb with the same orientation can be generated. Then, the operator can adjust the orientation, position, and size of the light box 3D annotation box and / or the bulb 3D annotation box through the three views, and project them onto the image domain in real time during the adjustment process to view the annotation quality.

[0081] In some examples, when the time-series data includes a sequence of images to be labeled, multiple frames of images to be labeled within the sequence can be identified. The generated second annotation result can then be projected onto each frame of the image to be labeled, allowing the user to verify the accuracy of the generated second annotation result. If the verification result indicates that the accuracy of the generated second annotation result does not meet the requirements, the generated second annotation result can be manually adjusted on the user interface until the projection meets the requirements. As shown in Figure 5C, image 52 shows the projection result of the 3D bulb annotation box in the second annotation result onto one image to be labeled, and image 53 shows the projection result of the 3D bulb annotation box in the second annotation result onto another image to be labeled. Furthermore, it can be seen from images 52 and 53 that the annotation quality of the 3D bulb annotation box in the 3D point cloud data projected onto the 2D (2D) bulb annotation box 54 in the image to be labeled meets the requirements (e.g., the fit between the 2D bulb annotation box 54 and the bulb of the traffic light in the image to be labeled meets the requirements), thus verifying that the annotation accuracy of the 3D bulb annotation box in the 3D point cloud data meets the requirements. It should be noted that Figures 52 and 53 include multiple projected 2D bounding boxes, but Figure 5C only shows a portion of the 2D bounding boxes as an example.

[0082] The target marker annotation method provided in this embodiment determines the position information of the prediction box and the type of target marker corresponding to the selection operation in response to the detection of a selection operation on the user interface when the target information is a prediction box of the predicted position. Based on the position information of the prediction box and the type of target marker, a second annotation result of the target marker in the three-dimensional point cloud data is generated. In this way, the second annotation result can be generated based on the predicted position of the target marker and the user's selection operation, thereby conveniently, quickly and accurately annotating the target marker.

[0083] As shown in Figure 6, based on the embodiment shown in Figure 2A above, step S202 may include the following steps S2021-S2024.

[0084] Step S2021: Construct three-dimensional point cloud data of the target scene based on time-series data, and perform semantic segmentation processing on the three-dimensional point cloud data to obtain the semantic information of the three-dimensional point cloud data.

[0085] For example, semantic segmentation can be performed on the 3D point cloud data of the target scene to obtain the semantic information of each point in the 3D point cloud data. For instance, a trained semantic segmentation model can be used to process the 3D point cloud data to obtain an image of the same size as the input point cloud, where each pixel corresponds to a semantic label of a point in the input point cloud. Commonly used semantic segmentation algorithms include PointNet, PointNet++, or Pillar-based methods.

[0086] It should be noted that the specific implementation method of semantic segmentation in this embodiment is not limited. Furthermore, there are no restrictions on aspects such as the model structure of the semantic segmentation model.

[0087] In some examples, the semantic information of 3D point cloud data refers to the content category of each point cloud in the 3D point cloud. For example, the semantic information of point cloud A in 3D point cloud data is a traffic light, the semantic information of point cloud B is a tree, the semantic information of point cloud C is a building, and the semantic information of point cloud D is a pedestrian, etc.

[0088] It is understood that the specific implementation method for constructing the 3D point cloud data of the target scene based on time-series data can be found in the description of step S202 above, and will not be repeated here.

[0089] Step S2022: Perform meshing processing on the 3D point cloud data to obtain multiple point cloud blocks.

[0090] For example, the three-dimensional point cloud data can be meshed to obtain multiple point cloud blocks; wherein each point cloud block represents a small area in space, containing a portion of the point cloud.

[0091] It should be noted that the granularity of the meshing process, i.e., the size of the point cloud blocks, is not limited in the embodiments of this disclosure. Those skilled in the art can set it according to actual usage requirements and the size of the 3D point cloud data.

[0092] Step S2023: Based on the semantic information of the 3D point cloud data, determine one or more target point cloud blocks from multiple point cloud blocks.

[0093] For example, if the target marker in this embodiment of the present disclosure is a traffic light, the point cloud block whose semantic information includes a traffic light can be determined as the target point cloud block from multiple point cloud data blocks based on the semantic information of the three-dimensional point cloud data. For instance, when the target marker is a traffic light, the semantic information of each point cloud can be compared with preset semantic information (traffic light) to obtain the semantic comparison result of each point cloud. Then, based on the semantic comparison result of each point cloud and the point cloud block to which the point cloud belongs, the target point cloud block is determined.

[0094] Step S2024: Perform clustering processing on one or more target point cloud blocks to obtain the predicted location of the target markers in the 3D point cloud data.

[0095] For example, after identifying one or more target point cloud blocks from multiple point cloud blocks, clustering can be performed on the one or more target point cloud blocks to obtain the predicted location of the target marker in the three-dimensional point cloud data.

[0096] In some examples, clustering multiple target point cloud blocks in this disclosure refers to the process of dividing multiple target point cloud blocks into several groups or clusters based on similarity. After the division is completed, a group or cluster is a target marker, and the predicted location of each target marker in the 3D point cloud data can be obtained.

[0097] The target marker annotation method provided in this embodiment of the present disclosure performs semantic segmentation processing on three-dimensional point cloud data to obtain semantic information of the three-dimensional point cloud data; performs meshing processing on the three-dimensional point cloud data to obtain multiple point cloud blocks; determines one or more target point cloud blocks from the multiple point cloud blocks based on the semantic information of the three-dimensional point cloud data; and performs clustering processing on the one or more target point cloud blocks to obtain the predicted position of the target marker in the three-dimensional point cloud data. In this way, the position of the target marker in the three-dimensional point cloud can be obtained by segmenting and clustering the three-dimensional point cloud data, and the position can be used to verify the pre-brushing results or to use the position as an anchor point to draw annotation information.

[0098] In some embodiments, determining one or more target point cloud blocks from multiple point cloud blocks based on the semantic information of 3D point cloud data includes: determining the point cloud included in each point cloud block and the semantic information of the point cloud included in each point cloud block; matching the semantic information of the point cloud included in each point cloud block with target markers to obtain the semantic matching result of each point cloud block; and determining the point cloud block with a successful semantic matching result as the target point cloud block.

[0099] For example, the semantic information of each point cloud in a point cloud block can be matched with the target marker to obtain the semantic matching result of each point cloud. Then, the point cloud block containing the point cloud with successful semantic matching is determined as the target point cloud block. In this way, the target point cloud block can be determined based on semantic information, thereby determining the predicted location of the target marker.

[0100] As shown in Figure 7, based on the embodiment shown in Figure 2A above, the following steps S206-S207 are also included.

[0101] Step S206: If the time series data includes a sequence of images to be labeled, determine multiple frames of images to be labeled in the sequence of images to be labeled.

[0102] For example, as shown in FIG1, vehicle 10 passes through target scene 13 at a preset driving speed, and image sensor 11 on vehicle 10 acquires images of target scene 13 at a preset frequency to obtain a sequence of images to be labeled. Then, multiple frames of images to be labeled in the sequence of images to be labeled can be determined.

[0103] Step S207: Project the second annotation result of the target marker in the 3D point cloud data onto each frame of the image to be annotated to obtain the annotation information of the target marker in each frame of the image to be annotated.

[0104] For example, the second annotation result (such as a 3D bounding box) of the target marker in the 3D point cloud data can be projected onto each frame of the image to be annotated to obtain the annotation information (such as a 2D bounding box) of the target marker in each frame of the image to be annotated. For instance, based on the position information of the target marker in the 3D point cloud data, the corresponding pixel of the target marker in the image to be annotated can be determined by coordinate system transformation. Then, the second annotation result of the target marker in the 3D point cloud can be projected onto the corresponding pixel in the image to be annotated to complete the annotation of the target marker on the image to be annotated.

[0105] The target marker annotation method provided in this embodiment determines multiple frames of images to be annotated in a sequence of images to be annotated; projects the second annotation result of the target marker in the three-dimensional point cloud data onto each frame of images to be annotated, thereby obtaining the annotation information of the target marker in each frame of images to be annotated; thus, the annotation of the target marker in the three-dimensional point cloud data constructed from time-series data can be achieved by projection, thereby obtaining the annotation result of the target marker in multiple images to be annotated in the target scene, thereby effectively improving the annotation efficiency of images.

[0106] Exemplary device

[0107] Figure 8 shows a target marker marking device provided in an embodiment of the present disclosure. As shown in Figure 8, the marking device 800 includes a time series data determination module 801, a predicted position determination module 802, a pre-brush result determination module 803, a marking display module 804, and a marking result determination module 805.

[0108] The timing data determination module 801 is used to determine the timing data corresponding to the target scene;

[0109] The prediction location determination module 802 is used to construct three-dimensional point cloud data of the target scene based on time-series data and determine the predicted location of the target marker in the three-dimensional point cloud data;

[0110] The pre-brush result determination module 803 is used to obtain the first annotation result of the target marker in the 3D point cloud data;

[0111] The annotation display module 804 is used to determine the target information corresponding to the target marker based on the predicted location and the first annotation result, and to display the target information on the user interface;

[0112] The annotation result determination module 805 is used to determine the second annotation result of the target marker in the 3D point cloud data in response to the detection of the annotation operation of the target information on the user interface.

[0113] In some embodiments, as shown in FIG9, the annotation display module 804 includes a matching unit 8041, a determining unit 8042, and a display unit 8043.

[0114] The matching unit 8041 is used to match the prediction box of the predicted location with the label box of the first labeling result to obtain the matching result;

[0115] The determining unit 8042 is used to determine the target information corresponding to the target marker based on the matching result;

[0116] Display unit 8043 is used to display target information on the user interface.

[0117] In some embodiments, the determining unit 8042 is specifically configured to determine the label box as target information corresponding to the target marker if the matching result shows that the predicted box and the label box are in the same position; or, if the matching result shows that the predicted box and the label box are not in the same position, determine the predicted box as target information corresponding to the target marker.

[0118] In some embodiments, when the target information is the annotation box of the first annotation result, the annotation result determination module 805 is specifically used to determine the adjusted first sub-annotation box in response to detecting the adjustment operation of the first sub-annotation box in the annotation box on the user interface; generate a second sub-annotation box based on the adjusted first sub-annotation box; and determine the second annotation result of the target marker in the three-dimensional point cloud data based on the second sub-annotation box and the adjusted first sub-annotation box.

[0119] In some embodiments, when the target information is a predicted bounding box of the predicted location, the annotation result determination module 805 is specifically used to determine the location information of the predicted bounding box and the type of the target marker corresponding to the selection operation in response to the detection of a selection operation on the user interface; and generate a second annotation result of the target marker in the three-dimensional point cloud data based on the location information of the predicted bounding box and the type of the target marker.

[0120] In some embodiments, the predicted location determination module 802 is used to construct three-dimensional point cloud data of the target scene based on time-series data; perform semantic segmentation processing on the three-dimensional point cloud data to obtain semantic information of the three-dimensional point cloud data; perform meshing processing on the three-dimensional point cloud data to obtain multiple point cloud blocks; determine one or more target point cloud blocks from the multiple point cloud blocks based on the semantic information of the three-dimensional point cloud data; and perform clustering processing on the one or more target point cloud blocks to obtain the predicted location of the target marker in the three-dimensional point cloud data.

[0121] In some embodiments, the predicted location determination module 802 is specifically used to construct three-dimensional point cloud data of the target scene based on time-series data; perform semantic segmentation processing on the three-dimensional point cloud data to obtain semantic information of the three-dimensional point cloud data; perform meshing processing on the three-dimensional point cloud data to obtain multiple point cloud blocks; determine the point clouds included in each point cloud block and the semantic information of the point clouds included in each point cloud block; match the semantic information of the point clouds included in each point cloud block with the target marker to obtain the semantic matching result of each point cloud block; determine the point cloud block with a successful semantic matching result as the target point cloud block; and perform clustering processing on one or more target point cloud blocks to obtain the predicted location of the target marker in the three-dimensional point cloud data.

[0122] In some embodiments, as shown in FIG9, when the time sequence data includes a sequence of images to be labeled, the labeling device 800 further includes a frame image determination module 806 and a projection module 807.

[0123] The frame image determination module 806 is used to determine multiple frames of images to be labeled in the image sequence to be labeled.

[0124] The projection module 807 is used to project the second annotation result of the target marker in the three-dimensional point cloud data onto each frame of the image to be annotated, so as to obtain the annotation information of the target marker in each frame of the image to be annotated.

[0125] The beneficial technical effects corresponding to the exemplary embodiment of the target marker labeling device 800 described above can be found in the corresponding beneficial technical effects of the exemplary method section described above, and will not be repeated here.

[0126] Exemplary electronic devices

[0127] Figure 10 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this disclosure, including at least one processor 101 and a memory 102 for storing processor-executable instructions. The processor 101 is used to read and execute executable instructions from the memory 102, and the processor 101 is configured to:

[0128] Determine the time-series data corresponding to the target scenario;

[0129] Construct 3D point cloud data of the target scene based on time-series data, and determine the predicted location of target landmarks in the 3D point cloud data;

[0130] Obtain the first annotation result of the target marker in the 3D point cloud data;

[0131] Based on the predicted location and the first annotation result, the target information corresponding to the target marker is determined and displayed on the user interface;

[0132] In response to the detection of annotation operations on target information on the user interface, a second annotation result of the target marker in the 3D point cloud data is determined.

[0133] In some embodiments, the processor determines the target information corresponding to the target marker based on the predicted location and the first annotation result. Specifically, it is configured to: match the predicted bounding box of the predicted location with the annotation bounding box of the first annotation result to obtain a matching result; and determine the target information corresponding to the target marker based on the matching result.

[0134] In some embodiments, the processor determines the target information corresponding to the target marker based on the matching result, specifically configured to: if the matching result shows that the predicted box and the labeled box are in the same position, the labeled box is determined as the target information corresponding to the target marker; or, if the matching result shows that the predicted box and the labeled box are not in the same position, the predicted box is determined as the target information corresponding to the target marker.

[0135] In some embodiments, when the target information is the bounding box of the first annotation result, the processor, in response to detecting an annotation operation on the target information on the user interface, determines the second annotation result of the target marker in the 3D point cloud data. Specifically, it is configured to: in response to detecting an adjustment operation on the first sub-bounding box of the annotation box on the user interface, determine the adjusted first sub-bounding box; generate a second sub-bounding box based on the adjusted first sub-bounding box; and determine the second annotation result of the target marker in the 3D point cloud data based on the second sub-bounding box and the adjusted first sub-bounding box.

[0136] In some embodiments, when the target information is a predicted bounding box of the predicted location, the processor, in response to detecting a labeling operation on the target information on the user interface, determines a second labeling result of the target marker in the 3D point cloud data. Specifically, it is configured to: in response to detecting a selection operation on the user interface, determine the position information of the predicted bounding box and the type of the target marker corresponding to the selection operation; and generate a second labeling result of the target marker in the 3D point cloud data based on the position information of the predicted bounding box and the type of the target marker.

[0137] In some embodiments, the processor determines the predicted location of a target marker in the 3D point cloud data by: performing semantic segmentation on the 3D point cloud data to obtain semantic information of the 3D point cloud data; performing meshing on the 3D point cloud data to obtain multiple point cloud blocks; determining multiple target point cloud blocks from the multiple point cloud blocks based on the semantic information of the 3D point cloud data; and performing clustering on the multiple target point cloud blocks to obtain the predicted location of the target marker in the 3D point cloud data.

[0138] In some embodiments, the processor determines multiple target point cloud blocks from multiple point cloud blocks based on the semantic information of the three-dimensional point cloud data. Specifically, the processor is configured to: determine the point cloud included in each point cloud block and the semantic information of the point cloud included in each point cloud block; match the semantic information of the point cloud included in each point cloud block with the target marker to obtain the semantic matching result of each point cloud block; and determine the point cloud block with a successful semantic matching result as the target point cloud block.

[0139] In some embodiments, when the time series data includes a sequence of images to be labeled, the processor is further configured to: determine multiple frames of images to be labeled in the sequence of images to be labeled; project the second labeling result of the target marker in the three-dimensional point cloud data onto each frame of images to be labeled, thereby obtaining the labeling information of the target marker in each frame of images to be labeled.

[0140] The processor 101 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0141] The memory 102 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 101 may execute one or more computer program instructions to implement the labeling methods for target markers and / or other desired functions of the various embodiments of this disclosure described above.

[0142] In one example, as shown in FIG10, the electronic device 100 may further include an input device 103 and an output device 104, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0143] The input device 103 may also include, for example, a keyboard, a mouse, etc.

[0144] The output device 104 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0145] Of course, for simplicity, Figure 10 only shows some of the components of the electronic device 100 that are relevant to this disclosure, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 100 may include any other suitable components depending on the specific application.

[0146] It is understood that the specific manner in which the processor performs the operation and the corresponding beneficial effects of the electronic device in the above embodiments have been described in detail in the corresponding embodiments of the aforementioned target marker labeling method section. Please refer to the corresponding execution operation method and beneficial technical effects in the above exemplary method section, which will not be repeated here.

[0147] Exemplary computer program products and computer-readable storage media

[0148] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the labeling methods for target markers described in the various embodiments of this disclosure in the "Exemplary Methods" section above.

[0149] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0150] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the labeling methods for target markers described in the various embodiments of this disclosure in the "Exemplary Methods" section above.

[0151] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0152] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0153] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An electronic device comprising a processor and a memory for storing processor-executable instructions; the processor being configured to read from and execute the executable instructions from the memory, the processor being configured to: Determine the time-series data corresponding to the target scenario; Based on the time-series data, a three-dimensional point cloud of the target scene is constructed, and the predicted position of the target landmark in the three-dimensional point cloud data is determined. Obtain the first annotation result of the target marker in the three-dimensional point cloud data; Based on the predicted location and the first annotation result, the target information corresponding to the target marker is determined and the target information is displayed on the user interface; In response to detecting a labeling operation on the target information on the user interface, a second labeling result for the target marker in the 3D point cloud data is determined. 2.The electronic device of claim 1, wherein, The processor determines the target information corresponding to the target marker based on the predicted location and the first annotation result, specifically configured as follows: The prediction box of the predicted location is matched with the annotation box of the first annotation result to obtain the matching result; Based on the matching results, the target information corresponding to the target marker is determined.

3. The electronic device of claim 2, wherein, Based on the matching result, the processor determines the target information corresponding to the target marker, specifically configured as follows: If the matching result indicates that the predicted bounding box and the labeled bounding box are in the same position, the labeled bounding box is determined as the target information corresponding to the target marker; or, If the matching result indicates that the predicted bounding box and the labeled bounding box are not in the same position, the predicted bounding box is determined as the target information corresponding to the target marker.

4. The electronic device of claim 3, wherein, When the target information is the bounding box of the first annotation result, the processor, in response to detecting the annotation operation on the target information on the user interface, determines the second annotation result of the target marker in the 3D point cloud data, specifically configured as follows: In response to detecting an adjustment operation on the first sub-label box of the label box on the user interface, the adjusted first sub-label box is determined; Based on the adjusted first sub-label box, generate the second sub-label box; Based on the second sub-label box and the adjusted first sub-label box, the second labeling result of the target marker in the three-dimensional point cloud data is determined.

5. The electronic device of claim 3, wherein, When the target information is a predicted bounding box of the predicted location, the processor, in response to detecting a labeling operation on the target information on the user interface, determines a second labeling result of the target marker in the 3D point cloud data, specifically configured as follows: In response to detecting a selection operation on the user interface, the position information of the prediction box and the type of the target marker corresponding to the selection operation are determined; Based on the location information of the prediction box and the type of the target marker, a second annotation result of the target marker in the 3D point cloud data is generated.

6. The electronic device of any of claims 1-5, wherein, The processor determines the predicted location of the target marker in the 3D point cloud data, specifically configured as follows: The semantic segmentation process is performed on the three-dimensional point cloud data to obtain the semantic information of the three-dimensional point cloud data. The three-dimensional point cloud data is processed into a grid to obtain multiple point cloud blocks; Based on the semantic information of the three-dimensional point cloud data, multiple target point cloud blocks are determined from the multiple point cloud blocks; Clustering is performed on the multiple target point cloud blocks to obtain the predicted positions of target markers in the three-dimensional point cloud data.

7. The electronic device of claim 6, wherein, The processor, based on the semantic information of the 3D point cloud data, determines multiple target point cloud blocks from the multiple point cloud blocks, specifically configured as follows: Determine the point cloud included in each of the plurality of point cloud blocks, and the semantic information of the point cloud included in each of the point cloud blocks; The semantic information of the point cloud included in each point cloud block is matched with the target marker to obtain the semantic matching result of each point cloud block. The point cloud block whose semantic matching result is successful is identified as the target point cloud block.

8. The electronic device according to any one of claims 1-5, wherein when the time-series data includes a sequence of images to be labeled, the processor is further configured to: Identify multiple frames of images to be labeled in the image sequence to be labeled; The second annotation result of the target marker in the three-dimensional point cloud data is projected onto each frame of the image to be annotated to obtain the annotation information of the target marker in each frame of the image to be annotated.

9. A method for labeling target markers, comprising: Determine the time-series data corresponding to the target scenario; Based on the time-series data, a three-dimensional point cloud of the target scene is constructed, and the predicted position of the target landmark in the three-dimensional point cloud data is determined. Obtain the first annotation result of the target marker in the three-dimensional point cloud data; Based on the predicted location and the first annotation result, the target information corresponding to the target marker is determined and the target information is displayed on the user interface; In response to detecting a labeling operation on the target information on the user interface, a second labeling result for the target marker in the 3D point cloud data is determined.

10. The method of claim 9, wherein, The step of determining the target information corresponding to the target marker based on the predicted location and the first annotation result includes: The prediction box of the predicted location is matched with the annotation box of the first annotation result to obtain the matching result; Based on the matching results, the target information corresponding to the target marker is determined.

11. The method of claim 10, wherein, The step of determining the target information corresponding to the target marker based on the matching result includes: If the matching result indicates that the predicted bounding box and the labeled bounding box are in the same position, the labeled bounding box is determined as the target information corresponding to the target marker; or, If the matching result indicates that the predicted bounding box and the labeled bounding box are not in the same position, the predicted bounding box is determined as the target information corresponding to the target marker.

12. The method of claim 11, wherein, When the target information is the annotation box of the first annotation result, the step of determining the second annotation result of the target marker in the 3D point cloud data in response to detecting the annotation operation of the target information on the user interface includes: In response to detecting an adjustment operation on the first sub-label box of the label box on the user interface, the adjusted first sub-label box is determined; Based on the adjusted first sub-label box, generate the second sub-label box; Based on the second sub-label box and the adjusted first sub-label box, the second labeling result of the target marker in the three-dimensional point cloud data is determined.

13. The method of claim 11, wherein, When the target information is a predicted bounding box of the predicted location, the step of determining the second annotation result of the target marker in the 3D point cloud data in response to detecting the annotation operation of the target information on the user interface includes: In response to detecting a selection operation on the user interface, the position information of the prediction box and the type of the target marker corresponding to the selection operation are determined; Based on the location information of the prediction box and the type of the target marker, a second annotation result of the target marker in the 3D point cloud data is generated.

14. The method of any one of claims 9-13, wherein, Determining the predicted location of the target marker in the 3D point cloud data includes: The semantic segmentation process is performed on the three-dimensional point cloud data to obtain the semantic information of the three-dimensional point cloud data. The three-dimensional point cloud data is processed into a grid to obtain multiple point cloud blocks; Based on the semantic information of the three-dimensional point cloud data, multiple target point cloud blocks are determined from the multiple point cloud blocks; Clustering is performed on the multiple target point cloud blocks to obtain the predicted positions of target markers in the three-dimensional point cloud data.

15. The method of claim 14, wherein, The semantic information based on the three-dimensional point cloud data is used to determine multiple target point cloud blocks from the multiple point cloud blocks, including: Determine the point cloud included in each of the plurality of point cloud blocks, and the semantic information of the point cloud included in each of the point cloud blocks; The semantic information of the point cloud included in each point cloud block is matched with the target marker to obtain the semantic matching result of each point cloud block. The point cloud block whose semantic matching result is successful is identified as the target point cloud block.

16. The method according to any one of claims 9-13, wherein when the time-series data includes a sequence of images to be labeled, the method further comprises: Identify multiple frames of images to be labeled in the image sequence to be labeled; The second annotation result of the target marker in the three-dimensional point cloud data is projected onto each frame of the image to be annotated to obtain the annotation information of the target marker in each frame of the image to be annotated.

17. A device for marking target markers, comprising: The time series data determination module is used to determine the time series data corresponding to the target scenario; The predicted location determination module is used to construct three-dimensional point cloud data of the target scene based on the time series data, and determine the predicted location of the target marker in the three-dimensional point cloud data; The pre-brush result determination module is used to obtain the first annotation result of the target marker in the three-dimensional point cloud data; The annotation display module is used to determine the target information corresponding to the target marker based on the predicted location and the first annotation result, and to display the target information on the user interface; The annotation result determination module is used to determine a second annotation result of the target marker in the three-dimensional point cloud data in response to detecting the annotation operation of the target information on the user interface. 18.A computer readable storage medium, the storage medium storing a computer program, the computer program being configured to perform the target marker labeling method according to any one of claims 9-16.