Video annotation method and apparatus, device, and storage medium

Abstract

A video annotation method, comprising: by means of key frame matching, automatically determining a key frame comprising a target object from a video to be annotated, and segmenting said video by using the key frame as a video segmentation point; and separately performing target tracking processing on two image sequences located before and after the key frame in the video. Thus, without changing the working principle of the target tracking algorithm, the video annotation method does not need to separately determine whether the first frame in the video to be annotated comprises the target object, thereby achieving the automation of video annotation and effectively improving the efficiency and accuracy of video annotation.

Description

A video annotation method, apparatus, device, and storage medium

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411554570.5, filed on November 1, 2024, entitled “A video annotation method, apparatus, device and storage medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of image processing technology, and more specifically, to a video annotation method, apparatus, device, and storage medium. Background Technology

[0004] In scenarios such as excavator operation and loader material handling, it is often necessary to record the working process of the vehicles in the scene on video. However, in addition to the target objects related to the video shooting task, such as the working vehicles and the materials being shoveled, the video often contains background elements unrelated to the video shooting task, such as trees and warehouses. Therefore, in order to improve the data quality of the video data, users need to label the above-mentioned target objects contained in the video data so that they can more intuitively and quickly locate the labeled target objects from the labeled video.

[0005] Currently, target tracking algorithms can be used to track specified objects in videos and mark their positions within each frame. However, since these algorithms typically only work on image sequences where the target object appears in the first frame and the images were captured consecutively, they cannot track video data where the target object does not appear in the first frame. This hinders the automation of video annotation, resulting in reduced efficiency and accuracy. Summary of the Invention

[0006] According to one aspect of this disclosure, a video annotation method is provided, comprising: determining, based on standard keyframes of a target object, image frames matching the standard keyframes from a video to be annotated as keyframes of the target object in the video to be annotated; wherein the standard keyframes include position marker information of the target object; combining the standard keyframes, the keyframes, and a first image sequence to obtain a first sub-video; and combining the standard keyframes, the keyframes, and a second image sequence to obtain a second sub-video; wherein the first image sequence represents an image sequence obtained by reversing image frames in the video to be annotated that precede the keyframes; and the second image sequence represents an image sequence composed of image frames in the video to be annotated that follow the keyframes; performing target tracking processing on the first sub-video and the second sub-video respectively, using the target object as the tracking target, to obtain target tracking processing results corresponding to the first sub-video and the second sub-video respectively; and recombining the target tracking processing results corresponding to the first sub-video and the second sub-video respectively according to the order of the image frames in the video to be annotated to obtain target tracking results of the target object in the video to be annotated.

[0007] According to one aspect of this disclosure, a video annotation apparatus is provided, comprising: a matching module, configured to determine, based on standard keyframes of a target object, image frames matching the standard keyframes from a video to be annotated as keyframes of the target object in the video to be annotated; wherein the standard keyframes include position marker information of the target object; a grouping module, configured to combine the standard keyframes, keyframes, and a first image sequence to obtain a first sub-video, and combine the standard keyframes, keyframes, and a second image sequence to obtain a second sub-video; wherein the first image sequence represents an image sequence obtained by reversing image frames in the video to be annotated that precede the keyframes; and the second image sequence represents an image sequence composed of image frames in the video to be annotated that follow the keyframes; a target tracking module, configured to perform target tracking processing on the first sub-video and the second sub-video respectively, using the target object as the tracking target, to obtain target tracking processing results corresponding to the first sub-video and the second sub-video respectively; and a recombination module, configured to recombine the target tracking processing results corresponding to the first sub-video and the second sub-video respectively, based on the order of image frames in the video to be annotated, to obtain target tracking results of the target object in the video to be annotated.

[0008] According to one aspect of this disclosure, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the video annotation method described above.

[0009] According to one aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the video annotation method described above.

[0010] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0011] This disclosure provides a video annotation method, apparatus, device, and storage medium that automatically determines keyframes containing target objects from a video to be annotated by keyframe matching. These keyframes are then used as video segmentation points to segment the video. Target tracking processing is performed on two sets of image sequences located before and after the keyframes in the video. This eliminates the need to separately determine whether the first frame of the video contains the target object without altering the underlying algorithm's working principle, thus automating video annotation and effectively improving its efficiency and accuracy. Attached Figure Description

[0012] Figure 1 shows a flowchart of one of the video annotation methods provided in this embodiment of the present disclosure;

[0013] Figure 2 shows a schematic diagram of the structure of one of the first sub-videos and the second sub-videos provided in an embodiment of this disclosure;

[0014] Figure 3 shows a flowchart illustrating one method for determining standard keyframes provided in an embodiment of this disclosure;

[0015] Figure 4 shows a flowchart illustrating one of the methods for deduplicating target tracking results of multiple target objects in a video to be labeled, provided by an embodiment of this disclosure.

[0016] Figure 5 shows a schematic diagram of the structure of one of the video annotation devices provided in the embodiments of this disclosure;

[0017] Figure 6 is a schematic diagram of the structure of one of the electronic devices provided in the embodiments of this disclosure. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this disclosure are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this disclosure. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this disclosure illustrate operations implemented according to some embodiments of this disclosure. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this disclosure, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0019] Furthermore, the described embodiments are merely some, not all, of the embodiments of this disclosure. The components of the embodiments of this disclosure typically described and illustrated in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the drawings is not intended to limit the scope of the claimed disclosure, but merely to illustrate selected embodiments of the disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0020] It should be noted that the term "comprising" will be used in the embodiments of this disclosure to indicate the presence of the features subsequently declared, but does not exclude the addition of other features.

[0021] Currently, target tracking algorithms can be used to track specified objects in videos and mark their positions within each frame. However, since these algorithms typically only work on image sequences where the target object appears in the first frame and the images were captured consecutively, they cannot track video data where the target object does not appear in the first frame. This hinders the automation of video annotation, resulting in reduced efficiency and accuracy.

[0022] Based on this, the present disclosure provides a video annotation method, apparatus, device, and storage medium. By using keyframe matching, keyframes containing target objects are automatically determined from the video to be annotated. The video to be annotated is then segmented using the keyframes as video segmentation points. Target tracking processing is performed on two sets of image sequences in the video to be annotated, one before the keyframe and the other after the keyframe. Thus, without changing the working principle of the target tracking algorithm, the video annotation method does not need to separately determine whether the first frame of the video to be annotated contains the target object, achieving automation of video annotation and effectively improving the efficiency and accuracy of video annotation.

[0023] In one embodiment of this disclosure, a video annotation method can run on a terminal device or a server. The terminal device can be a local terminal device. When the video annotation method runs on a server, it can be implemented and executed based on a cloud interaction system, which includes a server and client devices (i.e., terminal devices).

[0024] To facilitate understanding of the embodiments of this disclosure, a video annotation method, apparatus, device, and storage medium provided in the embodiments of this disclosure will be described in detail below.

[0025] Referring to Figure 1, which illustrates a flowchart of a video annotation method provided in this embodiment of the present disclosure, the video annotation method includes steps S101-S104; specifically:

[0026] S101, Based on the standard keyframes of the target object, determine the image frames that match the standard keyframes from the video to be labeled as the keyframes of the target object in the video to be labeled.

[0027] S102, combine the standard keyframe, the keyframe and the first image sequence to obtain the first sub-video, and combine the standard keyframe, the keyframe and the second image sequence to obtain the second sub-video.

[0028] S103, using the target object as the tracking target, perform target tracking processing on the first sub-video and the second sub-video respectively, and obtain the target tracking processing results corresponding to the first sub-video and the second sub-video respectively.

[0029] S104. Based on the order of the image frames in the video to be labeled, the target tracking processing results corresponding to the first sub-video and the second sub-video are recombined to obtain the target tracking result of the target object in the video to be labeled.

[0030] The video annotation method provided in this embodiment automatically determines the keyframes containing the target object from the video to be annotated by keyframe matching, and uses the keyframes as video segmentation points to segment the video to be annotated. Target tracking processing is then performed on two sets of image sequences in the video to be annotated, one before the keyframe and the other after the keyframe. This eliminates the need to separately determine whether the first frame of the video to be annotated contains the target object without changing the working principle of the target tracking algorithm, thus automating video annotation and effectively improving the efficiency and accuracy of video annotation.

[0031] The following is an exemplary description of each step in the video annotation method provided in the embodiments of this disclosure:

[0032] S101, Based on the standard keyframes of the target object, determine the image frames that match the standard keyframes from the video to be labeled as the keyframes of the target object in the video to be labeled.

[0033] Here, the target object refers to the entity object that needs to be labeled in the video to be labeled. The specific type of entity object represented by the target object can be determined according to the video shooting scene of the video to be labeled. The target object can represent only one entity object, or it can include multiple entity objects of different categories. This disclosure does not limit the specific type of entity object represented by the target object or the number of specific entity objects.

[0034] For example, taking the video shooting scene of the video to be labeled as an excavator operation scene as an example, the above-mentioned target object can be the excavator body (including the boom and bucket), or the excavator body, the material to be loaded (such as a pile of soil, a pile of materials, etc.), the truck transporting the material, etc.

[0035] Here, the above-mentioned keyframe annotation includes the location marker information of the target object. That is, the above-mentioned standard keyframe can be obtained by marking the target object in the image frame containing the target object.

[0036] It should be noted that the aforementioned standard keyframes may or may not come from the video to be labeled. For example, based on the video shooting scene of the video to be labeled, multiple videos from that shooting scene can be obtained. By selecting the image frame containing the target object from the multiple videos and marking the target object contained in the image frame, the aforementioned standard image frame can be obtained. The standard image frame obtained in this way is applicable to the video labeling of target objects in all videos to be labeled taken in the same video shooting scene, thereby improving the efficiency of video labeling.

[0037] Here, considering that the aforementioned standard keyframes may not belong to the video to be labeled, when performing step S101, the image frame with the highest degree of matching with the aforementioned standard keyframes can be determined from the video to be labeled by feature matching as the keyframe of the target object in the video to be labeled (i.e., the image frame that matches the aforementioned standard keyframes).

[0038] Specifically, as an optional embodiment, step S101 can be performed using ORB (Oriented Fast and Rotated BRIEF, a feature detection and description algorithm) feature matching. In the ORB feature matching method, for each image frame in the video to be labeled, the ORB algorithm can first detect key points in the image frame, and then calculate a feature vector (i.e., the ORB feature corresponding to the key point) for each key point, thus obtaining multiple ORB features corresponding to the image frame. Similarly, multiple ORB features corresponding to the standard key frame can also be obtained through the ORB algorithm. When the Euclidean distance between ORB features is less than a preset threshold, it can be determined that two ORB features match. The more ORB features that match, the higher the degree of matching between the image frame and the standard key frame. Thus, the image frame with the highest degree of matching with the standard key frame can be determined from the video to be labeled as the key frame of the target object in the video to be labeled.

[0039] It should be noted that the feature matching method that can be used when performing step S101 is not unique. This disclosure does not limit the specific implementation of the above feature matching method.

[0040] S102, combine the standard keyframe, the keyframe and the first image sequence to obtain the first sub-video, and combine the standard keyframe, the keyframe and the second image sequence to obtain the second sub-video.

[0041] Here, the first image sequence mentioned above represents the image sequence obtained by reversing the image frames in the video to be labeled that are located before the keyframes.

[0042] Specifically, Figure 2 shows a schematic diagram of the structure of a first sub-video and a second sub-video provided in an embodiment of this disclosure. As shown in Figure 2, the video to be labeled is segmented using the keyframes determined in step S101 as video segmentation points. Multiple image frames in the video to be labeled that are located before the keyframes can be determined (starting from the first frame in the video to be labeled shown in Figure 2, up to the frame before the keyframe). Considering that target tracking algorithms can usually only perform target tracking processing on image sequences where the target object appears in the first frame and the shooting time is continuous, when executing step S102, after placing the standard keyframe containing the target object and the keyframe in front, it is also necessary to reverse the multiple image frames in the video to be labeled that are located before the keyframes to obtain a first image sequence whose shooting time is continuous with the keyframes. Thus, according to the arrangement order of the standard keyframes, the keyframes and the first image sequence, the first sub-video (i.e., the video that can be target tracked by the target tracking algorithm) composed of the standard keyframes, the keyframes and the first image sequence is obtained.

[0043] Here, the second image sequence mentioned above represents an image sequence consisting of image frames located after the keyframe in the video to be annotated.

[0044] Specifically, as shown in Figure 2, after segmenting the video to be labeled using the keyframes determined in step S101 as video segmentation points, multiple image frames located after the keyframes in the video to be labeled (starting from the frame after the keyframe until the last frame in the video to be labeled) can also be determined. Considering that target tracking algorithms can usually only perform target tracking processing on image sequences where the target object appears in the first frame and the shooting time is continuous, when executing step S102, after placing the standard keyframe containing the target object and the keyframe at the front, the multiple image frames located after the keyframe that are continuous with the shooting time of the keyframe can be directly used as the second image sequence. Thus, according to the arrangement order of the standard keyframe, the keyframe, and the second image sequence, the second sub-video (i.e., the video that can be target tracked by the target tracking algorithm) composed of the standard keyframe, the keyframe, and the second image sequence is obtained.

[0045] S103, using the target object as the tracking target, perform target tracking processing on the first sub-video and the second sub-video respectively, and obtain the target tracking processing results corresponding to the first sub-video and the second sub-video respectively.

[0046] Here, when performing target tracking processing on the target objects contained in the first sub-video using the target tracking algorithm, based on the position marking information of the target objects contained in the standard keyframes, the target tracking algorithm can use the standard keyframes located at the foremost position in the first sub-video as prompt information related to the tracking target (equivalent to clearly identifying the target objects marked in the standard keyframes as the tracking targets). Based on this prompt information, the algorithm performs target tracking processing on the keyframes and the first image sequence that are captured in consecutive time in the first sub-video, and obtains the target tracking processing result corresponding to the first sub-video (i.e., the first sub-video marked with the position information of the target objects in each image frame of the first sub-video).

[0047] Here, when performing target tracking processing on the aforementioned target object contained in the second sub-video using the target tracking algorithm, the target tracking algorithm can also use the aforementioned standard keyframe located at the foremost position in the second sub-video as cue information related to the tracking target. Based on this cue information, target tracking processing is performed on the aforementioned keyframes and the aforementioned second image sequence that are captured in consecutive time in the second sub-video, to obtain the target tracking processing result corresponding to the second sub-video (i.e., the second sub-video marked with the position information of the target object in each image frame of the second sub-video).

[0048] It should be noted that the above target tracking algorithm can be the TrackAnything algorithm, or other target tracking algorithms that can only perform target tracking processing on image sequences in which the target object appears in the first frame and the shooting time is continuous. This disclosure does not limit the specific algorithm represented by the above target tracking algorithm.

[0049] S104. Based on the order of the image frames in the video to be labeled, the target tracking processing results corresponding to the first sub-video and the second sub-video are recombined to obtain the target tracking result of the target object in the video to be labeled.

[0050] Here, referring to the content of steps S102-S103 above, it can be seen that since the first image sequence in the first sub-video is the image sequence obtained by reversing the image frames located before the keyframe in the video to be labeled, after obtaining the target tracking processing results corresponding to the first sub-video and the second sub-video respectively, it is also necessary to adjust the arrangement order of each image frame in the target tracking processing results corresponding to the first sub-video and the second sub-video respectively according to the sorting of the image frames in the video to be labeled, so as to obtain the labeled video with the same sorting as the video to be labeled and containing the position marking information of the target object as the target tracking result of the target object in the video to be labeled.

[0051] In this embodiment of the disclosure, in conjunction with the above description, step S104 can be executed according to the method shown in steps a1-a4 below, specifically:

[0052] Step a1: Obtain the target tracking processing result of the first image sequence from the target tracking processing result corresponding to the first sub-video.

[0053] Here, referring to Figure 2, since the video to be labeled does not contain the above-mentioned standard keyframes, and the above-mentioned keyframes appear in both the first sub-video and the second sub-video, the target tracking processing result of the first image sequence (i.e., the first image sequence marked with the position information of the target object in each image frame of the first image sequence) can be obtained from the target tracking processing result corresponding to the first sub-video.

[0054] Step a2: Obtain the target tracking processing result of the second image sequence from the target tracking processing result corresponding to the second sub-video.

[0055] Here, referring to Figure 2, since the video to be labeled does not contain the above-mentioned standard keyframes, and the above-mentioned keyframes appear in both the first sub-video and the second sub-video, the target tracking processing result of the second image sequence (i.e., the second image sequence marked with the position information of the target object in each image frame of the second image sequence) can be obtained from the target tracking processing result corresponding to the second sub-video.

[0056] Step a3: Obtain the target tracking processing results of the keyframes from the target tracking processing results corresponding to the first or second sub-video.

[0057] Here, referring to Figure 2, since the keyframes appear in both the first sub-video and the second sub-video, the target tracking processing results of the keyframes (i.e., keyframes marked with the position information of the target object in the keyframes) can be obtained from the target tracking processing results corresponding to the first sub-video, or the target tracking processing results of the keyframes can be obtained from the target tracking processing results corresponding to the second sub-video, as long as there is no duplicate acquisition.

[0058] Step a4: Based on the order of the image frames in the video to be labeled, adjust the order of the image frames in the target tracking processing results of the first image sequence, the target tracking processing results of the key frames, and the target tracking processing results of the second image sequence to obtain the target tracking results of the target object in the video to be labeled.

[0059] Here, referring to Figure 2, based on the order of the image frames in the video to be labeled, the labeled image frames in the target tracking processing result of the first image sequence can be reversed to obtain the target tracking processing result of the first image sequence after reversal. Then, according to the order of the target tracking processing result of the first image sequence after reversal, the target tracking processing result of the key frame, and the target tracking processing result of the second image sequence, the labeled video composed of the above three target tracking processing results is obtained (i.e., the target tracking result of the target object in the video to be labeled).

[0060] It should be noted that, regardless of whether there is one or multiple target objects, each target object (that is, each entity object included in the target object) in the video to be annotated can be annotated according to the annotation method shown in steps S101-S104. The repetitive parts will not be repeated here.

[0061] The specific implementation process of each of the above steps in this embodiment of the disclosure will be described in detail below:

[0062] Regarding the method for determining the standard keyframe in step S101 above, in an optional embodiment, Figure 3 shows a flowchart of a method for determining the standard keyframe provided by an embodiment of this disclosure. As shown in Figure 3, before executing step S101, the method includes steps S301-S302; specifically:

[0063] S301, Based on the video shooting scene of the video to be labeled, determine the target image frame containing the target object from multiple original videos corresponding to the video shooting scene.

[0064] Here, the original video mentioned above can be a historical video taken in the video shooting scene. For example, if the video shooting scene is an excavator working scene, the original video mentioned above can be a historical video of multiple excavation operations of the excavator taken within a certain time period.

[0065] Specifically, when there are multiple image frames containing the target object in the aforementioned multiple original videos, as an optional embodiment, the image frame containing the target object and displaying the target object most clearly and completely can be determined from the multiple original videos as the target image frame.

[0066] It should be noted that, in the embodiments of this disclosure, it is only necessary to ensure that the target image frame contains the target object. The embodiments of this disclosure do not impose any limitations on the specific selection criteria for the target image frame.

[0067] S302, input the target image frame into the image segmentation model, annotate the target objects contained in the target image frame through the image segmentation model, and output the annotated target image frame as the standard keyframe of the target object.

[0068] Here, the target image frame is input into the image segmentation model. The image segmentation model can use the specified target object as a segmentation cue, predict local image regions in the target image frame that belong to the same entity category as the segmentation cue, and generate a segmentation mask corresponding to each predicted local region as the segmentation mask corresponding to the target object (that is, to label the target object contained in the target image frame); where the segmentation mask corresponding to the target object is the position marking information of the target object in the target image frame.

[0069] It should be noted that the above image segmentation models include, but are not limited to, the SAM model (Segment Anything model), the Mask-RCNN model, etc.; the specific model structure of the above image segmentation models is not limited in this disclosure embodiment.

[0070] Based on the video annotation method shown in steps S101-S104 above, considering that when there are multiple target objects to be annotated in the video to be annotated (i.e., the target objects include multiple entity objects of different categories), since multiple target objects may appear in the same image frame, there may be an overlap of the position markers (i.e. the above segmentation mask) of multiple target objects in the same image frame.

[0071] Here, to solve the above problems, in an optional implementation, Figure 4 shows a flowchart of a method for deduplicating target tracking results of multiple target objects in a video to be labeled, provided by an embodiment of this disclosure. As shown in Figure 4, after executing step S104, the method includes steps S401-S402; specifically:

[0072] S401, when the target object includes multiple entity objects of different categories, based on the target tracking results corresponding to the multiple entity objects in the video to be labeled, determine whether there are abnormal image frames with overlapping position markers among the target tracking results of the multiple entity objects.

[0073] Here, when there are multiple target objects (i.e., the target objects include multiple entity objects of different categories), since the target tracking processing is performed on multiple different entity objects appearing in the same video to be labeled, the target tracking results corresponding to the multiple entity objects in the video to be labeled (i.e., the segmentation mask of each entity object in each image frame in the video to be labeled) can be used to determine if the segmentation masks (i.e., mask markers) of multiple (two or more) entity objects overlap in the same image frame. Then, the image frame where the segmentation masks overlap is the above-mentioned abnormal image frame, and the overlapping segmentation mask is the local image region where the position markers overlap in the abnormal image frame.

[0074] S401, if it is determined that there is an abnormal image frame, perform segmentation prediction on the local image region where the position markers overlap in the abnormal image frame, and determine the entity objects to which different pixel units in the local image region belong based on the segmentation prediction results.

[0075] Here, the method for segmentation prediction of local image regions (i.e., overlapping segmentation masks) with overlapping position markers in abnormal image frames is not unique, and this disclosure does not limit it in any way.

[0076] Specifically, based on the fact that the aforementioned local image region (i.e., the overlapping segmentation mask) contains multiple sub-image regions enclosed by region boundaries (i.e., in the overlapping segmentation mask, multiple local regions enclosed by line segments often appear), in the first optional embodiment, for the multiple sub-image regions contained in the aforementioned local image region, according to the region boundary enclosing the sub-image region, the entity object whose position marker is connected to the region boundary is determined as the segmentation prediction result of the sub-image region.

[0077] For example, taking a target object including an excavator body and a pile of excavated material as an example, if the segmentation masks of the excavator body and the pile of material overlap in the same image frame, the image frame can be determined to be the aforementioned abnormal image frame. The segmentation mask of the overlapping part is determined to be the local image region of the aforementioned position mark overlap. At this time, for each local region (i.e., each sub-image region) enclosed by line segments in the segmentation mask of the overlapping part, it can be determined whether the region boundary (i.e., the line segment enclosing the local region) of the local region is connected to the segmentation mask of the excavator body or the segmentation mask of the pile of material. If the region boundary of the local region is connected to the segmentation mask of the excavator body, it can be determined that the local region belongs to the segmentation mask of the excavator body. If the region boundary of the local region is connected to the segmentation mask of the pile of material, it can be determined that the local region belongs to the segmentation mask of the pile of material. If there is no connection, the local region can be removed and the next local region can be determined.

[0078] Specifically, in the second optional implementation, the above-mentioned local image region (i.e., the overlapping segmentation mask) can be input into the image segmentation model by means of an image segmentation model, and the entity objects contained in the local image region can be segmented and predicted by the image segmentation model to obtain the segmentation prediction result of the local image region.

[0079] Here, the aforementioned local image regions (i.e., overlapping segmentation masks) are input into the image segmentation model. Based on the known information that multiple overlapping entity objects are involved in the aforementioned local image regions, the image segmentation model can use the aforementioned known information as segmentation cues. It can predict image regions from the local image regions that belong to the same entity category as each entity object involved in the aforementioned segmentation cues, and generate a segmentation mask corresponding to each predicted image region as the segmentation prediction result for that image region.

[0080] Specifically, as another optional embodiment, the segmentation prediction results of the local image regions with overlapping location markers can be comprehensively determined by combining the two optional implementation methods of the above-mentioned segmentation prediction, according to the following steps b1-b4:

[0081] Step b1: For multiple sub-image regions contained in a local image region, based on the region boundary that encloses the sub-image region, determine the entity object whose position marker is connected to the boundary of the region as the segmentation prediction result of the sub-image region.

[0082] Here, the specific implementation of step b1 is the same as the first optional implementation described above, and the repetitions will not be repeated here.

[0083] Step b2: Input the local image region into the image segmentation model, and use the image segmentation model to segment and predict the entity objects contained in the local image region to obtain the segmentation prediction result of the local image region.

[0084] Here, the specific implementation of step b2 is the same as the second optional implementation described above, and the repetitions will not be repeated here.

[0085] Step b3: If the segmentation prediction result of the sub-image region matches the segmentation prediction result of the local image region, then determine the entity object to which the sub-image region belongs based on the segmentation prediction result of the sub-image region.

[0086] Here, if the segmentation prediction result of the sub-image region is consistent with (i.e., matches) the segmentation prediction result of the local image region, then the entity object to which the sub-image region belongs can be directly determined as the segmentation prediction result of the sub-image region.

[0087] Step b4: If the segmentation prediction result of the sub-image region does not match the segmentation prediction result of the local image region, then remove the sub-image region from the local image region.

[0088] Here, if the segmentation prediction result of the sub-image region is inconsistent with the segmentation prediction result of the local image region (i.e., mismatch), the sub-image region is removed from the local image region (equivalent to removing the sub-image region whose entity object is uncertain).

[0089] Based on the video annotation method provided in this disclosure, keyframe matching is used to automatically determine keyframes containing target objects from the video to be annotated. The keyframes are then used as video segmentation points to segment the video to be annotated. Target tracking processing is performed on two sets of image sequences located before and after the keyframes in the video to be annotated. Thus, without changing the working principle of the target tracking algorithm, the video annotation method does not need to separately determine whether the first frame of the video to be annotated contains the target object, achieving automation of video annotation and effectively improving the efficiency and accuracy of video annotation.

[0090] Based on the same inventive concept, this disclosure also provides a video annotation device corresponding to the above-described video annotation method. Since the principle of solving the problem by the video annotation device in this disclosure is similar to that of the above-described video annotation method in this disclosure, the implementation of the video annotation device can refer to the implementation of the above-described video annotation method, and the repeated parts will not be described again.

[0091] Referring to FIG5, FIG5 shows a schematic diagram of the structure of a video annotation device provided in an embodiment of the present disclosure, wherein the video annotation device includes a matching module 501, a grouping module 502, a target tracking module 503 and a recombination module 504.

[0092] The matching module 501 is configured to determine, based on the standard keyframes of the target object, image frames that match the standard keyframes from the video to be labeled as keyframes of the target object in the video to be labeled; wherein the standard keyframes include the location marker information of the target object; the grouping module 502 is configured to combine the standard keyframes, keyframes and a first image sequence to obtain a first sub-video, and combine the standard keyframes, keyframes and a second image sequence to obtain a second sub-video; wherein the first image sequence represents the image sequence obtained by reversing the image frames in the video to be labeled that are located before the keyframes; the second image sequence represents the image sequence composed of the image frames in the video to be labeled that are located after the keyframes; the target tracking module 503 is configured to perform target tracking processing on the first sub-video and the second sub-video respectively, using the target object as the tracking target, to obtain the target tracking processing results corresponding to the first sub-video and the second sub-video respectively; the recombination module 504 is configured to recombine the target tracking processing results corresponding to the first sub-video and the second sub-video respectively according to the order of the image frames in the video to be labeled, to obtain the target tracking result of the target object in the video to be labeled.

[0093] In an optional implementation, the video annotation device further includes a standard keyframe determination module, wherein the standard keyframe determination module is configured to perform: based on the video shooting scene of the video to be annotated, determine the target image frame containing the target object from multiple original videos corresponding to the video shooting scene; input the target image frame into an image segmentation model, annotate the target object contained in the target image frame through the image segmentation model, and output the annotated target image frame as the standard keyframe of the target object.

[0094] In this embodiment, by inputting the target image frame into the image segmentation model, the image segmentation model can use the specified target object as a segmentation cue, predict local image regions in the target image frame that belong to the same entity category as the segmentation cue, and generate a segmentation mask corresponding to each predicted local region as the segmentation mask corresponding to the target object. This is the position marking information of the target object in the target image frame, thereby accurately obtaining the labeled target image frame as the standard keyframe of the target object. This facilitates the subsequent automation of video annotation and effectively improves the efficiency and accuracy of video annotation.

[0095] In an optional implementation, when recombining the target tracking processing results corresponding to the first sub-video and the second sub-video according to the order of image frames in the video to be labeled, the recombining module 504 is specifically configured to perform the following: obtaining the target tracking processing results of the first image sequence from the target tracking processing results corresponding to the first sub-video; obtaining the target tracking processing results of the second image sequence from the target tracking processing results corresponding to the second sub-video; obtaining the target tracking processing results of the keyframes from the target tracking processing results corresponding to the first sub-video or the second sub-video; and adjusting the order of image frames in the target tracking processing results of the first image sequence, the target tracking processing results of the keyframes, and the target tracking processing results of the second image sequence according to the order of image frames in the video to be labeled, thereby obtaining the target tracking results of the target object in the video to be labeled.

[0096] In an optional implementation, the video annotation device further includes a deduplication module, wherein the deduplication module is configured to perform: when the target object includes multiple entity objects of different categories, based on the target tracking results corresponding to the multiple entity objects in the video to be annotated, determine whether there are abnormal image frames with overlapping position markers among the target tracking results of the multiple entity objects; if it is determined that there are abnormal image frames, perform segmentation prediction on the local image region with overlapping position markers in the abnormal image frame, and determine the entity objects to which different pixel units in the local image region belong based on the segmentation prediction results.

[0097] The above embodiments can solve the problem that when there are multiple target objects to be labeled in a video, the position markers (i.e., the segmentation mask) of multiple target objects may overlap in the same image frame because multiple target objects may appear in the same image frame. This realizes the automation of video annotation and effectively improves the efficiency and accuracy of video annotation.

[0098] In one optional implementation, when performing segmentation prediction on local image regions with overlapping location markers in an abnormal image frame, a deduplication module is configured to execute: for multiple sub-image regions contained in a local image region, based on the region boundary enclosing the sub-image region, determine the entity object whose location marker is connected to the boundary of that region as the segmentation prediction result of the sub-image region.

[0099] In one optional implementation, when performing segmentation prediction on local image regions with overlapping location markers in abnormal image frames, a deduplication module is configured to execute: inputting the local image region into an image segmentation model, and using the image segmentation model to perform segmentation prediction on the entity objects contained in the local image region, thereby obtaining the segmentation prediction result of the local image region.

[0100] In one optional implementation, when performing segmentation prediction on local image regions with overlapping location markers in an abnormal image frame, a deduplication module is configured to execute: for multiple sub-image regions contained in the local image region, based on the region boundary enclosing the sub-image region, determine the entity object whose location marker is connected to the boundary of that region as the segmentation prediction result of the sub-image region; input the local image region into the image segmentation model, and perform segmentation prediction on the entity objects contained in the local image region through the image segmentation model to obtain the segmentation prediction result of the local image region; if the segmentation prediction result of the sub-image region matches the segmentation prediction result of the local image region, then determine the entity object to which the sub-image region belongs based on the segmentation prediction result of the sub-image region; if the segmentation prediction result of the sub-image region does not match the segmentation prediction result of the local image region, then remove the sub-image region from the local image region.

[0101] Based on the video annotation apparatus provided in this embodiment, the keyframe containing the target object is automatically determined from the video to be annotated by keyframe matching. The keyframe is then used as the video segmentation point to segment the video to be annotated. Target tracking processing is performed on the two sets of image sequences located before and after the keyframe in the video to be annotated. Thus, without changing the working principle of the target tracking algorithm, the video annotation method does not need to separately determine whether the first frame of the video to be annotated contains the target object, thereby achieving automation of video annotation and effectively improving the efficiency and accuracy of video annotation.

[0102] Based on the same inventive concept, this disclosure also provides an electronic device corresponding to the above-described video annotation method. Since the principle of solving the problem by the electronic device in the embodiments of this disclosure is similar to that of the above-described video annotation method in the embodiments of this disclosure, the implementation of the electronic device can refer to the implementation of the above-described video annotation method, and the repeated parts will not be described again.

[0103] Figure 6 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of this disclosure, including: a processor 601, a memory 602, and a bus 603. The memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device runs a video annotation method as described in the embodiment, the processor 601 communicates with the memory 602 through the bus 603. The processor 601 executes the machine-readable instructions, wherein the processor 601 executes the following steps when executing the machine-readable instructions:

[0104] Based on the standard keyframes of the target object, image frames matching the standard keyframes are identified from the video to be labeled as keyframes of the target object in the video to be labeled; wherein, the standard keyframes include the location marker information of the target object; the standard keyframes, keyframes and a first image sequence are combined to obtain a first sub-video, and the standard keyframes, keyframes and a second image sequence are combined to obtain a second sub-video; wherein, the first image sequence represents the image sequence obtained by reversing the image frames in the video to be labeled that are located before the keyframes; the second image sequence represents the image sequence composed of the image frames in the video to be labeled that are located after the keyframes; using the target object as the tracking target, target tracking processing is performed on the first sub-video and the second sub-video respectively to obtain the target tracking processing results corresponding to the first sub-video and the second sub-video respectively; according to the order of the image frames in the video to be labeled, the target tracking processing results corresponding to the first sub-video and the second sub-video are recombined to obtain the target tracking result of the target object in the video to be labeled.

[0105] In an optional implementation, the processor 601 is further configured to: determine a target image frame containing a target object from multiple original videos corresponding to the video shooting scene based on the video shooting scene of the video to be labeled; input the target image frame into an image segmentation model, label the target object contained in the target image frame through the image segmentation model, and output the labeled target image frame as the standard keyframe of the target object.

[0106] In this embodiment, by inputting the target image frame into the image segmentation model, the image segmentation model can use the specified target object as a segmentation cue, predict local image regions in the target image frame that belong to the same entity category as the segmentation cue, and generate a segmentation mask corresponding to each predicted local region as the segmentation mask corresponding to the target object. This is the position marking information of the target object in the target image frame, thereby accurately obtaining the labeled target image frame as the standard keyframe of the target object. This facilitates the subsequent automation of video annotation and effectively improves the efficiency and accuracy of video annotation.

[0107] In an optional implementation, when recombining the target tracking processing results corresponding to the first sub-video and the second sub-video according to the order of image frames in the video to be labeled, the processor 601 is configured to: obtain the target tracking processing results of the first image sequence from the target tracking processing results corresponding to the first sub-video; obtain the target tracking processing results of the second image sequence from the target tracking processing results corresponding to the second sub-video; obtain the target tracking processing results of the keyframes from the target tracking processing results corresponding to the first sub-video or the second sub-video; and adjust the order of image frames in the target tracking processing results of the first image sequence, the target tracking processing results of the keyframes, and the target tracking processing results of the second image sequence according to the order of image frames in the video to be labeled, thereby obtaining the target tracking results of the target object in the video to be labeled.

[0108] In an optional implementation, the processor 601 is further configured to: when the target object includes multiple entity objects of different categories, determine whether there are abnormal image frames with overlapping position markers among the target tracking results of the multiple entity objects in the video to be labeled, based on the target tracking results corresponding to the multiple entity objects respectively; if it is determined that there are abnormal image frames, perform segmentation prediction on the local image region with overlapping position markers in the abnormal image frame, and determine the entity objects to which different pixel units in the local image region belong based on the segmentation prediction results.

[0109] The above embodiments can solve the problem that when there are multiple target objects to be labeled in a video, the position markers (i.e., the segmentation mask) of multiple target objects may overlap in the same image frame because multiple target objects may appear in the same image frame. This realizes the automation of video annotation and effectively improves the efficiency and accuracy of video annotation.

[0110] In an optional implementation, when performing segmentation prediction on local image regions where location markers overlap in an abnormal image frame, the processor 601 is configured to: for multiple sub-image regions contained in the local image region, determine the entity object whose location marker is connected to the boundary of the region as the segmentation prediction result of the sub-image region based on the region boundary that encloses the sub-image region.

[0111] In an optional implementation, when performing segmentation prediction on local image regions with overlapping location markers in an abnormal image frame, the processor 601 is configured to: input the local image region into an image segmentation model, perform segmentation prediction on the entity objects contained in the local image region through the image segmentation model, and obtain the segmentation prediction result of the local image region.

[0112] In an optional implementation, when performing segmentation prediction on local image regions with overlapping location markers in an abnormal image frame, the processor 601 is configured to: for multiple sub-image regions contained in the local image region, determine the entity object whose location marker is connected to the boundary of the region that encloses the sub-image region as the segmentation prediction result of the sub-image region based on the region boundary that encloses the sub-image region; input the local image region into an image segmentation model, and perform segmentation prediction on the entity object contained in the local image region through the image segmentation model to obtain the segmentation prediction result of the local image region; if the segmentation prediction result of the sub-image region matches the segmentation prediction result of the local image region, then determine the entity object to which the sub-image region belongs based on the segmentation prediction result of the sub-image region; if the segmentation prediction result of the sub-image region does not match the segmentation prediction result of the local image region, then remove the sub-image region from the local image region.

[0113] The electronic device provided in this embodiment automatically determines keyframes containing target objects from the video to be labeled by keyframe matching, and segments the video to be labeled using keyframes as video segmentation points. Target tracking processing is then performed on two sets of image sequences in the video to be labeled, one before the keyframe and the other after the keyframe. This eliminates the need to separately determine whether the first frame of the video to be labeled contains the target object without changing the working principle of the target tracking algorithm, thus automating video labeling and effectively improving the efficiency and accuracy of video labeling.

[0114] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium storing a computer program, which is executed by a processor, and the processor performs the following steps:

[0115] Based on the standard keyframes of the target object, image frames matching the standard keyframes are identified from the video to be labeled as keyframes of the target object in the video to be labeled; wherein, the standard keyframes include the location marker information of the target object; the standard keyframes, keyframes and a first image sequence are combined to obtain a first sub-video, and the standard keyframes, keyframes and a second image sequence are combined to obtain a second sub-video; wherein, the first image sequence represents the image sequence obtained by reversing the image frames in the video to be labeled that are located before the keyframes; the second image sequence represents the image sequence composed of the image frames in the video to be labeled that are located after the keyframes; using the target object as the tracking target, target tracking processing is performed on the first sub-video and the second sub-video respectively to obtain the target tracking processing results corresponding to the first sub-video and the second sub-video respectively; according to the order of the image frames in the video to be labeled, the target tracking processing results corresponding to the first sub-video and the second sub-video are recombined to obtain the target tracking result of the target object in the video to be labeled.

[0116] In an optional implementation, the processor is further configured to: determine a target image frame containing a target object from multiple original videos corresponding to the video shooting scene, based on the video shooting scene of the video to be labeled; input the target image frame into an image segmentation model, label the target object contained in the target image frame through the image segmentation model, and output the labeled target image frame as the standard keyframe of the target object.

[0117] In this embodiment, by inputting the target image frame into the image segmentation model, the image segmentation model can use the specified target object as a segmentation cue, predict local image regions in the target image frame that belong to the same entity category as the segmentation cue, and generate a segmentation mask corresponding to each predicted local region as the segmentation mask corresponding to the target object. This is the position marking information of the target object in the target image frame, thereby accurately obtaining the labeled target image frame as the standard keyframe of the target object. This facilitates the subsequent automation of video annotation and effectively improves the efficiency and accuracy of video annotation.

[0118] In one optional implementation, when recombining the target tracking processing results corresponding to the first sub-video and the second sub-video according to the order of image frames in the video to be labeled, the processor is configured to: obtain the target tracking processing results of the first image sequence from the target tracking processing results corresponding to the first sub-video; obtain the target tracking processing results of the second image sequence from the target tracking processing results corresponding to the second sub-video; obtain the target tracking processing results of the keyframes from the target tracking processing results corresponding to the first sub-video or the second sub-video; and adjust the order of image frames in the target tracking processing results of the first image sequence, the target tracking processing results of the keyframes, and the target tracking processing results of the second image sequence according to the order of image frames in the video to be labeled, thereby obtaining the target tracking results of the target object in the video to be labeled.

[0119] In an optional implementation, the processor is further configured to: when the target object includes multiple entity objects of different categories, determine whether there are abnormal image frames with overlapping position markers among the target tracking results of the multiple entity objects in the video to be labeled, based on the target tracking results corresponding to the multiple entity objects respectively; if it is determined that there are abnormal image frames, perform segmentation prediction on the local image region with overlapping position markers in the abnormal image frame, and determine the entity objects to which different pixel units in the local image region belong based on the segmentation prediction results.

[0120] The above embodiments can solve the problem that when there are multiple target objects to be labeled in a video, the position markers (i.e., the segmentation mask) of multiple target objects may overlap in the same image frame because multiple target objects may appear in the same image frame. This realizes the automation of video annotation and effectively improves the efficiency and accuracy of video annotation.

[0121] In one optional implementation, when performing segmentation prediction on local image regions with overlapping location markers in an abnormal image frame, the processor is configured to: for multiple sub-image regions contained in the local image region, determine the entity object whose location marker is connected to the boundary of the region as the segmentation prediction result of the sub-image region based on the region boundary that encloses the sub-image region.

[0122] In one optional implementation, when performing segmentation prediction on local image regions with overlapping location markers in an abnormal image frame, the processor is configured to: input the local image regions into an image segmentation model, perform segmentation prediction on the entity objects contained in the local image regions through the image segmentation model, and obtain the segmentation prediction result of the local image regions.

[0123] In one optional implementation, when performing segmentation prediction on local image regions with overlapping location markers in an abnormal image frame, the processor is configured to: for multiple sub-image regions contained in the local image region, determine the entity object whose location marker is connected to the boundary of the region that encloses the sub-image region as the segmentation prediction result of the sub-image region based on the region boundary that encloses the sub-image region; input the local image region into the image segmentation model, and perform segmentation prediction on the entity object contained in the local image region through the image segmentation model to obtain the segmentation prediction result of the local image region; if the segmentation prediction result of the sub-image region matches the segmentation prediction result of the local image region, then determine the entity object to which the sub-image region belongs based on the segmentation prediction result of the sub-image region; if the segmentation prediction result of the sub-image region does not match the segmentation prediction result of the local image region, then remove the sub-image region from the local image region.

[0124] By using the computer-readable storage medium provided in the embodiments of this disclosure, and through the keyframe matching method, keyframes containing target objects are automatically determined from the video to be labeled, and the video to be labeled is segmented using the keyframes as video segmentation points. Target tracking processing is then performed on two sets of image sequences in the video to be labeled, one before the keyframe and the other after the keyframe. Thus, without changing the working principle of the target tracking algorithm, the video labeling method does not need to separately determine whether the first frame of the video to be labeled contains the target object, thereby achieving automated video labeling and effectively improving the efficiency and accuracy of video labeling.

[0125] In this embodiment of the disclosure, the computer-readable storage medium can also execute other machine-readable instructions when the processor runs, so as to perform other video annotation methods as in the embodiment. For the specific video annotation method steps and principles, please refer to the description of the method-side embodiment, which will not be repeated here.

[0126] In the embodiments provided in this disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0127] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] In addition, the functional units in the embodiments provided in this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0129] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0130] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0131] Finally, it should be noted that the above embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure. All should be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A video annotation method, the video annotation method comprising: Based on the standard keyframes of the target object, image frames that match the standard keyframes are determined from the video to be labeled as keyframes of the target object in the video to be labeled; wherein, the standard keyframes include the location marker information of the target object; The standard keyframe, the keyframe, and the first image sequence are combined to obtain a first sub-video; the standard keyframe and the keyframe are combined to obtain a second sub-video; wherein, the first image sequence represents the image sequence obtained by reversing the image frames in the video to be labeled that are located before the keyframe; the second image sequence represents the image sequence composed of the image frames in the video to be labeled that are located after the keyframe. Using the target object as the tracking target, target tracking processing is performed on the first sub-video and the second sub-video respectively to obtain the target tracking processing results corresponding to the first sub-video and the second sub-video respectively; Based on the order of image frames in the video to be labeled, the target tracking processing results corresponding to the first sub-video and the second sub-video are recombined to obtain the target tracking result of the target object in the video to be labeled.

2. The video annotation method according to claim 1, wherein, The video annotation method also includes: Based on the video shooting scene of the video to be labeled, a target image frame containing the target object is determined from multiple original videos corresponding to the video shooting scene; The target image frame is input into the image segmentation model, and the target object contained in the target image frame is labeled by the image segmentation model. The labeled target image frame is then output as the standard keyframe of the target object.

3. The video annotation method according to claim 1, wherein, The step of reorganizing the target tracking processing results corresponding to the first sub-video and the second sub-video respectively according to the sorting of image frames in the video to be labeled includes: Obtain the target tracking processing result of the first image sequence from the target tracking processing result corresponding to the first sub-video; Obtain the target tracking processing result of the second image sequence from the target tracking processing result corresponding to the second sub-video; Obtain the target tracking processing result of the key frame from the target tracking processing result corresponding to the first sub-video or the second sub-video; Based on the order of the image frames in the video to be labeled, the order of the image frames in the target tracking processing results of the first image sequence, the target tracking processing results of the key frames, and the target tracking processing results of the second image sequence is adjusted to obtain the target tracking result of the target object in the video to be labeled.

4. The video annotation method according to claim 1, wherein, The video annotation method also includes: When the target object includes multiple entity objects of different categories, based on the target tracking results corresponding to the multiple entity objects in the video to be labeled, it is determined whether there are abnormal image frames with overlapping position markers among the target tracking results of the multiple entity objects. If the abnormal image frame is determined to exist, the local image region with overlapping position markers in the abnormal image frame is segmented and predicted, and the entity objects to which different pixel units in the local image region belong are determined based on the segmentation and prediction results.

5. The video annotation method according to claim 4, wherein, The segmentation and prediction of local image regions with overlapping location markers in the abnormal image frames includes: For multiple sub-image regions contained in the local image region, based on the region boundary that surrounds the sub-image region, the entity object whose position marker is connected to the boundary of the region is determined as the segmentation prediction result of the sub-image region.

6. The video annotation method according to claim 4, wherein, The segmentation and prediction of local image regions with overlapping location markers in the abnormal image frame further includes: The local image region is input into the image segmentation model, and the image segmentation model is used to segment and predict the entity objects contained in the local image region to obtain the segmentation prediction result of the local image region.

7. The video annotation method according to claim 4, wherein, The segmentation and prediction of local image regions with overlapping location markers in the abnormal image frame further includes: For multiple sub-image regions contained in the local image region, based on the region boundary that surrounds the sub-image region, the entity object whose position marker is connected to the boundary of the region is determined as the segmentation prediction result of the sub-image region; The local image region is input into the image segmentation model, and the image segmentation model is used to segment and predict the entity objects contained in the local image region to obtain the segmentation prediction result of the local image region. If the segmentation prediction result of the sub-image region matches the segmentation prediction result of the local image region, then the entity object to which the sub-image region belongs is determined based on the segmentation prediction result of the sub-image region. If the segmentation prediction result of the sub-image region does not match the segmentation prediction result of the local image region, then the sub-image region is removed from the local image region.

8. A video annotation device, the video annotation device comprising: The matching module is configured to perform operations based on standard keyframes of the target object, determining image frames that match the standard keyframes from the video to be labeled as keyframes of the target object in the video to be labeled; wherein, the standard keyframes include the location marker information of the target object; The grouping module is configured to combine the standard keyframe, the keyframe, and a first image sequence to obtain a first sub-video, and combine the standard keyframe, the keyframe, and a second image sequence to obtain a second sub-video; wherein, the first image sequence represents the image sequence obtained by reversing the image frames in the video to be labeled that are located before the keyframe; the second image sequence represents the image sequence composed of the image frames in the video to be labeled that are located after the keyframe; The target tracking module is configured to perform target tracking processing on the first sub-video and the second sub-video respectively, using the target object as the tracking target, to obtain the target tracking processing results corresponding to the first sub-video and the second sub-video respectively; The recombination module is configured to recombine the target tracking processing results corresponding to the first sub-video and the second sub-video respectively according to the sorting of image frames in the video to be labeled, so as to obtain the target tracking result of the target object in the video to be labeled.

9. An electronic device, comprising: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the video annotation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the video annotation method as described in any one of claims 1 to 7.

Images