Intelligent tracking photographing gimbal, photographing method, device, and storage medium

Abstract

An intelligent tracking photographing gimbal, a photographing method, an electronic device, and a storage medium. The intelligent tracking photographing gimbal comprises a gimbal base (1), a gimbal housing (2), a tracker (3) and a driving mechanism; the tracker and the driving mechanism are arranged in the gimbal housing; the tracker comprises an AI camera (31); the AI camera is used for acquiring a captured first picture comprising a key target. The photographing method comprises: inputting the first picture into a target detection model to obtain a key target position; then converting the key target position into first physical coordinates on the basis of a distortion parameter and a rotation angle of the AI camera, and then performing intelligent tracking photographing on the key target on the basis of the first physical coordinates and a multi-target tracking algorithm. The electronic device stores a computer program for implementing the photographing method. The storage medium stores a program capable of executing the photographing method.

Description

A smart tracking and shooting gimbal, shooting method, device and storage medium

[0001] This application claims priority to the following Chinese patent applications: application number 2025104198626, filed April 3, 2025, entitled "An intelligent tracking and shooting method, shooting gimbal, device and storage medium"; application number 2025104388584, filed April 9, 2025, entitled "A real-time tracking and shooting method, device, equipment and medium based on jersey number"; and application number 202423143844X, filed December 9, 2024, entitled "A tracking gimbal and tracking shooting device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of this application relate to the field of image recognition technology, and in particular to an intelligent tracking and shooting gimbal, shooting method, device and storage medium. Background Technology

[0003] Tracking technology is a crucial tool for capturing images in sports events, widely used in professional scenarios such as athlete motion capture and live event broadcasting. However, currently used video tracking methods still face significant technical bottlenecks in intelligent tracking and multi-angle acquisition in dynamic scenes. Firstly, traditional tracking pan-tilt systems often employ fixed-angle shooting or manual control, failing to enable autonomous shooting decisions in dynamic scenes and easily leading to the loss of multi-dimensional image information. Secondly, tracking algorithms generally employ the principle of prioritizing the largest target or feature recognition based on clothing color, only capable of tracking the entire group within the field. They cannot accurately distinguish individual athletes from specific teams, lack facial feature extraction and multimodal verification mechanisms, and are prone to target misjudgment or loss of tracking in scenes with multiple athletes wearing similar clothing, resulting in insufficient accuracy in dynamic tracking.

[0004] Furthermore, traditional personnel tracking algorithms rely on clothing color and appearance features for target association. In scenarios where team uniforms are similar in color and athletes are densely packed together, feature confusion can easily lead to broken tracking trajectories or misidentification, severely impacting the accuracy of data detection. Additionally, some studies have attempted to integrate features such as athletes' hairstyles and body types to improve robustness, but these complex multimodal models are bulky and difficult to run in real-time on low-computing-power mobile devices, limiting their application in scenarios such as live streaming and instant replay.

[0005] Furthermore, the driving device of the shooting gimbal in the existing technology is usually set on the clamp connected to the handle. The omnidirectional tracking is achieved by controlling the rotation of the clamp, which has a relatively complex driving structure. Moreover, the existing automatic shooting methods for sports events generally adopt panoramic stitching technology, which constructs a panoramic image by correcting distortion of multiple wide-angle lenses and stitching images together. The hardware storage cost is high, and it is difficult to meet the real-time requirements of live sports broadcasts.

[0006] Therefore, there is an urgent need for a real-time intelligent tracking and shooting method to meet the needs of intelligent, low-cost, and high-efficiency shooting of sports events. Summary of the Invention

[0007] According to embodiments of this application, an intelligent tracking and shooting gimbal and shooting method are provided, which can achieve efficient intelligent tracking and shooting, and significantly reduce the cost of shooting sports events.

[0008] In a first aspect of this application, an intelligent tracking and shooting gimbal is provided, including a gimbal base, a gimbal housing, a tracker, and a drive mechanism, wherein the tracker and the drive mechanism are disposed within the gimbal housing;

[0009] The tracker includes an AI camera, which is the tracking sensor in application number 202423143844X. The AI ​​camera is used to acquire a first image containing a key target, input the first image into a target detection model to obtain the position of the key target; convert the position of the key target into first physical coordinates according to the distortion parameters and rotation angle of the AI ​​camera; intelligently track the key target according to the first physical coordinates and a multi-target tracking algorithm, and send rotation commands to the drive mechanism.

[0010] The drive mechanism is used to receive the rotation command and drive the shooting gimbal to rotate; the drive mechanism includes a first drive mechanism and a second drive mechanism, the first drive mechanism is driven and connected to the gimbal base to realize the horizontal 360-degree rotation of the tracker; the second drive mechanism is driven and connected to the tracker to realize the vertical 90-degree tilt rotation of the tracker.

[0011] Optionally, key objectives may include, but are not limited to, people and the ball.

[0012] In one possible implementation, in addition to intelligently tracking and shooting key targets, it will continuously acquire images of the key targets' movement and analyze the regions of interest in the images, and send rotation commands to the shooting gimbal based on the movement of the regions of interest in the images.

[0013] Optionally, the selection method for the region of interest in the image includes, but is not limited to, user-defined selection and automatic selection by algorithm. In user-defined selection, users can manually paint or outline the area of ​​interest in the image through the interactive interface.

[0014] In one possible implementation, intelligent tracking and imaging of key targets is performed based on first physical coordinates and a multi-target tracking algorithm, including:

[0015] The multi-target tracking algorithm predicts the movement of the key target based on the first physical coordinates and obtains the second physical coordinates of the key target;

[0016] The camera automatically adjusts the pitch and yaw angles of the shooting gimbal based on the second physical coordinates to continuously acquire a second view containing the key target.

[0017] Optionally, it also includes a face recognition and face skeleton point detection model, a face feature extraction model, and a human body feature extraction model;

[0018] The face features of the first image are extracted using a face recognition and face skeleton point detection model and a face feature extraction model to obtain the first face feature data.

[0019] Human features are extracted from the first image using a human feature extraction model to obtain first human feature data.

[0020] The second image is sequentially input into the human feature extraction model for feature extraction to obtain the second human feature data. The third human feature data that is similar to the first human feature data is selected from the second human feature data, and intelligent target tracking and shooting are performed on the third image corresponding to the third human feature data.

[0021] The face features of the third image are extracted using a face recognition and face skeleton point detection model and a face feature extraction model to obtain the second face feature data. The third image is then reconfirmed based on the first face feature data and the second face feature data.

[0022] Optionally, facial features are extracted from the first image using a face recognition and facial skeleton point detection model and a face feature extraction model to obtain first facial feature data, including:

[0023] The face is located and aligned in the first image by using a face recognition and face skeleton point detection model to obtain the first face data and the first face skeleton point data.

[0024] The first face data and the first face skeleton data are input into the face feature extraction model to obtain the first face feature data.

[0025] Optionally, the third image is further confirmed based on the first facial feature data and the second facial feature data, including:

[0026] The first facial feature data and the second facial feature data are matched for similarity.

[0027] If the first facial feature data and the second facial feature data are similar, then continue with intelligent target tracking and shooting;

[0028] If the first facial feature data and the second facial feature data are not similar, then the third image will be re-filtered.

[0029] Optionally, when performing intelligent target tracking and shooting of key targets, a third frame is randomly selected and input into the human feature extraction model for feature extraction to obtain fourth human feature data.

[0030] Perform similarity matching between the fourth human feature data and the target feature queue;

[0031] If the fourth human feature data is similar to the target feature queue, then continue with intelligent target tracking and shooting;

[0032] If the fourth human feature data is not similar to the target feature queue, the intelligent target tracking shooting will fail, the intelligent target tracking shooting will be paused, and the third screen will be selected again.

[0033] The target feature queue is a set of feature sequences of all third-party human feature data collected in history.

[0034] In one possible implementation, the camera gimbal is communicatively connected to the mobile terminal, specifically:

[0035] Obtain user operation commands on the mobile terminal;

[0036] Adjust the pitch and yaw angles of the shooting gimbal according to the operation instructions, and zoom in and out of the selected image in the operation instructions on the mobile terminal.

[0037] In one possible implementation, the camera gimbal also includes a display screen, which can directly display the execution results captured by the camera gimbal after receiving the driving command.

[0038] In one possible implementation, the AI ​​camera captures the user's specified actions as the start of the operation command. The AI ​​camera only begins to recognize the user's operation command after the user completes the action command on the display screen within a specified time.

[0039] Furthermore, a clamping structure for holding the camera device is fixedly provided above the tracker.

[0040] Furthermore, the tracker also includes a tracking housing for fixing the AI ​​camera, and an arc-shaped rack is provided on one side of the tracking housing. The second drive mechanism is provided with a drive gear that is adapted to the arc-shaped rack.

[0041] Furthermore, a bearing seat is provided in the middle part of the gimbal base, and the driver of the first drive mechanism is rotatably fixedly connected to the bearing seat.

[0042] Furthermore, a control motherboard is provided between the driver of the first drive mechanism and the bearing seat. The control motherboard is fixedly connected to the driver of the first drive mechanism, and both the control motherboard and the driver of the first drive mechanism are fixedly connected to the gimbal housing.

[0043] Furthermore, a support assembly is also provided within the accommodating space of the gimbal housing. The support assembly includes a main support and an auxiliary support, with the auxiliary support disposed on both sides of the tracker.

[0044] Furthermore, the bracket is provided with a support bearing, and the tracker has protruding posts on both sides, which are rotatably and fixedly connected to the support bearing.

[0045] Furthermore, the AI ​​camera is mounted on one side of the main bracket, and a power supply is provided on the other side of the main bracket.

[0046] Furthermore, a switch button is provided on the gimbal housing.

[0047] In one possible implementation, the AI ​​camera is also adapted to intelligently track and capture jersey numbers, specifically including:

[0048] Acquire the first human image containing the key target;

[0049] The first human body image is input into the target detection model, the number area in the first human body image is identified and cropped to obtain the first number image;

[0050] Input the first jersey number image into the number classification model to obtain the first jersey number data;

[0051] Based on the first jersey number data, key targets are matched and tracked in real-time shooting footage.

[0052] Optionally, based on the first jersey number data, key targets are matched and tracked in the live-stream footage, including:

[0053] The continuous real-time captured images are input into the target detection model, which sequentially identifies and crops the number regions in the real-time captured images to obtain the second number image queue.

[0054] The second jersey number data queue is obtained by sequentially classifying the second number images in the second number image queue using a number classification model.

[0055] The system searches for a second jersey number that matches the first jersey number in the second jersey number data queue. If a second jersey number that matches the first jersey number exists, the system tracks and captures the first human target corresponding to the second jersey number using a target tracking algorithm. If no second jersey number that matches the first jersey number exists, the system continues to identify and match.

[0056] Optionally, a target tracking algorithm is used to track and capture images of the first human target corresponding to the second jersey number data, including:

[0057] The movement of the first human target is predicted by the target tracking algorithm to obtain the next predicted position of the first human target;

[0058] The camera automatically adjusts the pitch and yaw angles of the shooting gimbal based on the predicted position, continuously acquiring images of the second human body, including the first human target.

[0059] Optionally, the continuously acquired second human images are sequentially identified and classified using an object detection model and a number classification model to obtain a third jersey number data queue;

[0060] The third jersey number data in the third jersey number data queue is matched with the second jersey number data in turn. If the third jersey number data in the third jersey number data queue matches the second jersey number data, the tracking and shooting continues. If there is a third jersey number data in the third jersey number data queue that does not match the second jersey number data, the tracking and shooting fails, and the key target is matched again in the real-time shooting screen for tracking and shooting.

[0061] In one possible implementation, continuous real-time captured images are input into a target detection model to sequentially identify and crop the number regions in the real-time captured images, thereby obtaining a second number image queue. The implementation also includes:

[0062] When the live video is captured from a distance, the live video is divided into video blocks, and each video block is magnified by a preset magnification.

[0063] The number region is identified and cropped for each enlarged image block using an object detection model, resulting in a second number image queue.

[0064] Optionally, if during the tracking and shooting process, the target detection model and the number classification model identify that the fourth jersey number data on the second human target in the real-time shooting image is consistent with the second jersey number data, then the tracking target of the tracking and shooting is transferred to the second human target.

[0065] Optionally, the first human image and the first jersey number data can be added to the training data of the target detection model and the number classification model on the server side for model iteration.

[0066] In a second aspect of this application, an intelligent tracking and shooting method is provided. The method includes:

[0067] The system acquires the first image containing the key target captured by the AI ​​camera in the shooting gimbal, and inputs the first image into the target detection model to obtain the location of the key target.

[0068] The key target positions are converted into first physical coordinates based on the distortion parameters and rotation angle of the AI ​​camera;

[0069] Intelligent tracking and imaging of key targets are performed based on the first physical coordinates and a multi-target tracking algorithm.

[0070] Optionally, key objectives may include, but are not limited to, people and the ball.

[0071] In one possible implementation, in addition to intelligently tracking and capturing images of the key target, the system continuously acquires images of the key target's movement and analyzes the region of interest (ROI) within the image. Based on the movement of the ROI, rotation commands are sent to the pan-tilt unit.

[0072] Optionally, the selection method for the region of interest in the image includes, but is not limited to, user-defined selection and automatic selection by algorithm. In user-defined selection, users can manually paint or outline the area of ​​interest in the image through the interactive interface.

[0073] In one possible implementation, intelligent tracking and imaging of key targets is performed based on first physical coordinates and a multi-target tracking algorithm, including:

[0074] The multi-target tracking algorithm predicts the movement of the key target based on the first physical coordinates and obtains the second physical coordinates of the key target;

[0075] The camera automatically adjusts the pitch and yaw angles of the shooting gimbal based on the second physical coordinates to continuously acquire a second view containing the key target.

[0076] Optionally, the method further includes:

[0077] The face features of the first image are extracted using a face recognition and face skeleton point detection model and a face feature extraction model to obtain the first face feature data.

[0078] Human features are extracted from the first image using a human feature extraction model to obtain first human feature data.

[0079] The second image is sequentially input into the human feature extraction model for feature extraction to obtain the second human feature data. The third human feature data that is similar to the first human feature data is selected from the second human feature data, and intelligent target tracking and shooting are performed on the third image corresponding to the third human feature data.

[0080] The face features of the third image are extracted using a face recognition and face skeleton point detection model and a face feature extraction model to obtain the second face feature data. The third image is then reconfirmed based on the first face feature data and the second face feature data.

[0081] Optionally, facial features are extracted from the first image using a face recognition and facial skeleton point detection model and a face feature extraction model to obtain first facial feature data, including:

[0082] The face is located and aligned in the first image by using a face recognition and face skeleton point detection model to obtain the first face data and the first face skeleton point data.

[0083] The first face data and the first face skeleton data are input into the face feature extraction model to obtain the first face feature data.

[0084] Optionally, the third image is further confirmed based on the first facial feature data and the second facial feature data, including:

[0085] The first facial feature data and the second facial feature data are matched for similarity.

[0086] If the first facial feature data and the second facial feature data are similar, then continue with intelligent target tracking and shooting;

[0087] If the first facial feature data and the second facial feature data are not similar, then the third image will be re-filtered.

[0088] Optionally, the method further includes:

[0089] When intelligent target tracking and shooting of key targets, a third frame is randomly selected and input into the human feature extraction model for feature extraction to obtain the fourth human feature data.

[0090] Perform similarity matching between the fourth human feature data and the target feature queue;

[0091] If the fourth human feature data is similar to the target feature queue, then continue with intelligent target tracking and shooting;

[0092] If the fourth human feature data is not similar to the target feature queue, the intelligent target tracking shooting will fail, the intelligent target tracking shooting will be paused, and the third screen will be selected again.

[0093] The target feature queue is a set of feature sequences of all third-party human feature data collected in history.

[0094] In one possible implementation, the camera gimbal is communicatively connected to the mobile terminal, and the method further includes:

[0095] Obtain user operation commands on the mobile terminal;

[0096] Adjust the pitch and yaw angles of the shooting gimbal according to the operation instructions, and zoom in and out of the selected image in the operation instructions on the mobile terminal.

[0097] In one possible implementation, the camera gimbal also includes a display screen, which can directly display the execution results captured by the camera gimbal after receiving the driving command.

[0098] In one possible implementation, the AI ​​camera captures the user's specified actions as the start of the operation command. The AI ​​camera only begins to recognize the user's operation command after the user completes the action command on the display screen within a specified time.

[0099] In a third aspect of this application, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0100] In a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method as described in the second aspect of this application.

[0101] The advantages of this application are as follows:

[0102] The intelligent tracking and shooting gimbal provided in this application embodiment houses the tracker and the drive mechanism for rotating the tracker within the gimbal housing. The first drive mechanism enables 360-degree horizontal rotation of the tracker, while the second drive mechanism enables 90-degree vertical tilt rotation, achieving effective and accurate omnidirectional tracking. This design offers advantages such as simple structure, ease of use, and high stability and precision in gimbal control. Furthermore, by acquiring a first image containing a key target captured by the AI ​​camera within the gimbal, and inputting this image into a target detection model, the key target's position is obtained. Then, based on the AI ​​camera's distortion parameters and rotation angle, the key target's position is converted into first physical coordinates. Finally, based on these first physical coordinates and a multi-target tracking algorithm, intelligent tracking and shooting of the key target is performed, achieving highly efficient intelligent tracking and shooting, improving the efficiency of event shooting, and reducing shooting costs.

[0103] Furthermore, the intelligent tracking and shooting gimbal in this embodiment is also suitable for tracking and shooting jersey numbers. Specifically, it acquires a first human image containing a key target, inputs the first human image into a target detection model, identifies the number region in the first human image, and crops it to obtain a first number image. Then, the first number image is input into a number classification model to obtain first jersey number data. Finally, based on the first jersey number data, it matches the key target in the real-time shooting image and performs tracking and shooting, realizing real-time jersey number recognition and providing a foundation for subsequent tracking and shooting of matches.

[0104] It should be understood that the description in the Summary Section is not intended to limit the key or essential features of the embodiments of this application, nor is it intended to restrict the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0105] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0106] Figure 1 is a flowchart of an intelligent tracking and shooting method according to an embodiment of this application;

[0107] Figure 2 is a flowchart of intelligent tracking and shooting for key target confirmation according to an embodiment of this application;

[0108] Figure 3 is a schematic diagram of the interaction between the mobile terminal and the camera gimbal according to an embodiment of this application;

[0109] Figure 4 is a block diagram of an intelligent tracking and shooting gimbal device according to an embodiment of this application;

[0110] Figure 5 is a flowchart of an intelligent tracking and shooting method based on jersey number according to another embodiment of this application;

[0111] Figure 6 is a schematic diagram of an intelligent tracking and shooting method based on jersey number according to another embodiment of this application;

[0112] Figure 7 is a schematic diagram of jersey number recognition according to another embodiment of this application;

[0113] Figure 8 is a schematic diagram of the offline training terminal model training process according to another embodiment of this application;

[0114] Figure 9 is a block diagram of an intelligent tracking and shooting platform based on jersey number according to another embodiment of this application;

[0115] Figure 10 is a three-dimensional structural diagram of the camera gimbal in another embodiment of this application;

[0116] Figure 11 is a three-dimensional structural diagram of another embodiment of this application after removing the gimbal housing;

[0117] Figure 12 is a three-dimensional structural diagram of the tracking housing in another embodiment of this application;

[0118] Figure 13 is a schematic diagram of the structure of a terminal device or server suitable for implementing the embodiments of this application.

[0119] Reference numerals: 1. Gimbal base; 2. Gimbal housing; 3. Tracker; 31. AI camera; 32. Tracking housing; 41. First drive mechanism; 42. Second drive mechanism; 411. Bearing seat; 51. Main bracket; 52. Auxiliary bracket; 6. Clamping structure; 7. Switch button; 8. Power supply; 9. Control motherboard. Detailed Implementation

[0120] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0121] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0122] In this application, unless otherwise expressly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "connection" can refer to a fixed connection, a detachable connection, or an integral part, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0123] Furthermore, the technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of a person skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0124] Example 1

[0125] Figure 1 is a flowchart of an intelligent tracking and shooting method according to an embodiment of this application. Referring to Figure 1, the method includes:

[0126] The system acquires the first image containing the key target captured by the AI ​​camera in the shooting gimbal, and inputs the first image into the target detection model to obtain the location of the key target.

[0127] Key targets include, but are not limited to, people and balls in sports events. Users can customize the selection of key targets on mobile terminals (including but not limited to smartphones and tablets) through tapping, voice control, and gesture control. Then, the mobile terminal sends the key target information to the shooting gimbal.

[0128] The target detection model is used to identify and locate the category and location information of key targets from images and video streams. The target detection model used in this application includes, but is not limited to, the YOLO (You Only Look Once) model. By dynamically dividing the first frame image containing the key target into a variable density grid of 52×52 to 104×104, the grid granularity is automatically adjusted according to the initial size of the target to ensure the detection recall rate of small targets (such as 10×10 pixels). Then, multiple bounding boxes and class probabilities are predicted for each grid, thereby achieving end-to-end real-time detection.

[0129] In this embodiment, a target detection model is used to achieve accurate identification of key targets, which has anti-interference ability in complex scenes and effectively overcomes the problem of missed detection in traditional visual detection.

[0130] The location of key targets is converted into first physical coordinates based on the distortion parameters and rotation angle of the AI ​​camera.

[0131] In one possible implementation, the key target location is (u,v). First, the key target location is distorted, and the formula for calculating the distorted coordinates (u',v') is as follows:

[0132] r 2 =u d 2 +v d 2 ,

[0133] u c =u d (1+k1r 2 +k2r 4 +k3r 6 )+2p1u d v d +p2(r 2 +2u d 2 ),

[0134] v c =v d (1+k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2v d 2 )+2p2u d v d ,

[0135] u'=f x u c +c x ,v'=f y v c +c y ,

[0136] Among them, (c x ,c y ) represents the coordinates of the center point of the image, f x f y Let be the focal length of the AI ​​camera, k1, k2, and k3 be the radial distortion coefficients, and p1 and p2 be the tangential distortion coefficients. Then, the distortion-free coordinates (u', v') are normalized to obtain the normalized coordinates:

[0137]

[0138] Finally, rotate the normalized coordinates (x, y) to the direction of the first physical coordinate system. dir ,y dir ,z dir The calculation method for ) is as follows:

[0139] Where R is a preset rotation matrix mapped to the direction of the first physical coordinate system.

[0140] In this embodiment, a precise physical spatial coordinate system is established through dual processing of distortion correction and spatial coordinate mapping, eliminating the positioning deviation caused by AI camera distortion and providing standardized spatial data input for subsequent intelligent tracking and shooting.

[0141] Intelligent tracking and imaging of key targets are performed based on the first physical coordinates and a multi-target tracking algorithm.

[0142] In one possible implementation, in addition to intelligently tracking and capturing key targets, the system continuously acquires images of the target's movement and analyzes the region of interest (ROI) within the image. Based on the movement of the ROI, rotation commands are sent to the pan-tilt unit. Generating intelligent control rotation commands for the pan-tilt unit based on the analysis of the ROI eliminates the need for panoramic stitching or dynamic cropping to achieve tracking, thus reducing the cost of event filming while ensuring high-resolution capture.

[0143] The Region of Interest (ROI) is a local area in the image to be processed, delineated using rectangles, circles, ellipses, and irregular polygons. Methods for selecting ROIs include, but are not limited to, user-defined selection and automatic algorithm selection. In user-defined selection, users can manually draw or outline the ROI through the interactive interface. In automatic algorithm selection, the object detection model outputs an initial bounding box for the key object after the user selects it; this bounding box is the initial ROI. For example, if the user selects a running athlete in the image, the object detection model will generate a rectangular box covering the athlete's entire body; this box is the initial ROI. Then, optical flow analysis is used to analyze the spatiotemporal gradient changes between adjacent frames containing the key object to infer the movement of the key object, obtaining a new ROI that covers the location the key object might reach in the next frame, preventing the key object from moving out of the detection range. Among them, the optical flow method aims to improve the efficiency and accuracy of motion estimation through local computation. First, corner points or feature points with significant texture are extracted in the initial ROI region. Then, based on the gradient equation constructed by the assumption of constant brightness and the extracted feature points of the initial ROI region, the key feature points of the new ROI region are calculated, thereby obtaining the new ROI region.

[0144] Furthermore, based on the movement of the region of interest (ROI) in the image, rotation commands are sent to the camera gimbal to generate AI camera rotation commands that combine analysis of ROI position changes with physical space coordinate transformation. First, the center point (x, y) of the current frame's ROI is calculated. c ,y c The pixel offset between the initial ROI region center point (x0, y0) and the ROI region center point (x0, y0) is calculated using the formula: △x = x c-x0, △y=y c -y0. Then, based on the current pitch and yaw angles of the gimbal involved in the shooting within the camera, the pixel offset is converted into the angle that the gimbal needs to adjust. The calculation formula is as follows:

[0145] Where △φ is the calculated yaw angle that needs to be adjusted, △θ is the calculated pitch angle that needs to be adjusted, s is the pixel size of the AI ​​camera, f is the focal length of the AI ​​camera, and φ is the current initial yaw angle of the shooting gimbal.

[0146] In this embodiment, the multi-target tracking algorithm organically combines key target trajectory prediction with motion compensation, ensuring tracking continuity in complex scenarios.

[0147] Optionally, intelligent tracking and imaging of key targets are performed based on the first physical coordinates and a multi-target tracking algorithm, including:

[0148] The multi-target tracking algorithm predicts the movement of the key target based on the first physical coordinates and obtains the second physical coordinates of the key target;

[0149] The camera automatically adjusts the pitch and yaw angles of the shooting gimbal based on the second physical coordinates to continuously acquire a second view containing the key target.

[0150] Among them, the Simple Online and Realtime Tracking (SORT) multi-object tracking algorithm is a high-efficiency multi-object tracking algorithm based on motion modeling and data association. Its core idea is to achieve continuous tracking of target identity across frames by fusing target motion prediction in the temporal dimension and detection box position information in the spatial dimension. First, the position and bounding box of the key target detected by the target detection model in each frame are obtained. Then, Kalman filtering is used to predict the bounding box position of the key target in the next frame based on the historical trajectory (i.e., the first physical coordinates) of the key target. Finally, the Hungarian algorithm is introduced to establish the association between the current frame containing the key target and the previous frame containing the key target. By minimizing the association cost, the key target in the current frame is matched with the key target in the previous frame, ensuring that the bounding box of each key target is associated with at most one predicted bounding box, thereby obtaining the second physical coordinates of the key target.

[0151] In one possible implementation, the first physical coordinates are (x1, y1), and the second physical coordinates predicted by the multi-target tracking algorithm are (x2, y2). Then, the yaw angle Δφ that the gimbal needs to adjust is:

[0152] Where D represents the target horizontal parameter. The pitch angle Δθ that the shooting gimbal needs to adjust is:

[0153] In addition, the rotation speed of the shooting gimbal can be controlled by a PID controller. A smooth rotation speed command is generated based on the pitch angle difference or yaw angle difference to avoid gimbal shake. The formula for calculating the smooth rotation speed u(t) is as follows:

[0154] Where e(t) is the pitch angle difference or yaw angle difference, K p To set the hyperparameter to a value of 0.8, K i To set the hyperparameter to a value of 0.2, K d Set the hyperparameter to a value of 0.1.

[0155] In this embodiment, through deep collaboration between the multi-target tracking algorithm and the gimbal control in the shooting gimbal, highly robust intelligent tracking is achieved in complex sports shooting scenarios.

[0156] Optionally, the method further includes:

[0157] The face features of the first image are extracted using a face recognition and face skeleton point detection model and a face feature extraction model to obtain the first face feature data.

[0158] Human features are extracted from the first image using a human feature extraction model to obtain first human feature data.

[0159] The second image is sequentially input into the human feature extraction model for feature extraction to obtain the second human feature data. The third human feature data that is similar to the first human feature data is selected from the second human feature data, and intelligent target tracking and shooting are performed on the third image corresponding to the third human feature data.

[0160] The face features of the third image are extracted using a face recognition and face skeleton point detection model and a face feature extraction model to obtain the second face feature data. The third image is then reconfirmed based on the first face feature data and the second face feature data.

[0161] Figure 2 is a flowchart of intelligent tracking and shooting key target confirmation according to an embodiment of this application, as shown in Figure 2.

[0162] The face recognition and facial skeleton detection model employs the lightweight RetinaFace-MobileNet model to quickly locate the facial bounding boxes of key targets in the captured image. Then, based on the MediaPipe Face Mesh model, 468 facial key points (including but not limited to eyelids, corners of the mouth, and tip of the nose) are extracted to construct the facial topology of the key targets. Face feature extraction models include, but are not limited to, ArcFace, OpenFace, and Dlib, used to extract facial feature data from the image. Human body feature extraction models include, but are not limited to, CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), and LSTM (Long Short-Term Memory), used to extract human body feature data from the image. In this invention, when the shooting gimbal performs intelligent target tracking and shooting of a key target, it confirms the correct tracking of the key target by comparing the similarity of human feature data. At the same time, the shooting gimbal synchronizes the key target image captured by the shooting gimbal to the mobile terminal, and the mobile terminal confirms the correct tracking and shooting of the key target by comparing the similarity of facial feature data. This realizes the matching of key targets between the intelligent terminal and the shooting gimbal.

[0163] In this embodiment, combining both facial and human biometric features improves the accuracy of key target locking and tracking, avoiding interference from occlusion, clothing changes, etc. Furthermore, by initially screening human feature data and then matching it with facial feature data, the false tracking rate during shooting is reduced.

[0164] Optionally, facial features are extracted from the first image using a face recognition and facial skeleton point detection model and a face feature extraction model to obtain first facial feature data, including:

[0165] The face is located and aligned in the first image by using a face recognition and face skeleton point detection model to obtain the first face data and the first face skeleton point data.

[0166] The first face data and the first face skeleton data are input into the face feature extraction model to obtain the first face feature data.

[0167] In one possible implementation, if the face recognition and face skeleton point detection model cannot detect the first face data and the first face skeleton point data in the first frame, it continues to detect in subsequent frames of the first frame until the first face data and the first face skeleton point data of the key target are obtained, and then saves the first face data and the first face skeleton point data in the mobile terminal.

[0168] In this embodiment, geometric alignment is achieved through face recognition and face skeleton point detection models, which eliminates the interference of pose differences on face feature extraction and improves the discriminative power of the face feature data extracted by the face feature extraction model.

[0169] Optionally, the third image is further confirmed based on the first facial feature data and the second facial feature data, including:

[0170] The first facial feature data and the second facial feature data are matched for similarity.

[0171] If the first facial feature data and the second facial feature data are similar, then continue with intelligent target tracking and shooting;

[0172] If the first facial feature data and the second facial feature data are not similar, then the third image will be re-filtered.

[0173] For example, if the first facial feature data is [0.12, 0.34, 0.56, ... 0.78] and the second facial feature data is [0.13, 0.35, 0.57, ... 0.79], and the similarity matching value between the first and second facial feature data is calculated using Euclidean distance, it is 0.05. If the similarity threshold for facial feature data is set to 0.5, then the first and second facial feature data are not similar, and the AI ​​camera's tracking and shooting of the key target fails. It is necessary to compare the human feature data again in the captured image and re-select the third image.

[0174] In this embodiment, the key target is accurately identified by facial feature data similarity matching, ensuring the continuity of shooting and tracking in complex event scenarios.

[0175] Optionally, the method further includes:

[0176] When intelligent target tracking and shooting of key targets, a third frame is randomly selected and input into the human feature extraction model for feature extraction to obtain the fourth human feature data.

[0177] Perform similarity matching between the fourth human feature data and the target feature queue;

[0178] If the fourth human feature data is similar to the target feature queue, then continue with intelligent target tracking and shooting;

[0179] If the fourth human feature data is not similar to the target feature queue, the intelligent target tracking shooting will fail, the intelligent target tracking shooting will be paused, and the third screen will be selected again.

[0180] The target feature queue is a set of feature sequences of all third-party human feature data collected in history.

[0181] For example, the fourth human feature data is [0.1, 0.6, 0.8, ..., 0.9], and the target feature queue is [[0.1, 0.3, 0.5, ..., 0.2], ..., [0.2, 0.4, 0.7, ..., 0.3]]. The similarity value between the fourth human feature data and each feature data in the target feature queue is calculated sequentially using Euclidean distance. Then, all similarity values ​​are averaged. If the similarity value between the fourth human feature data and the target feature queue is 0.3, then the fourth human feature data and the target feature queue are not similar, and the AI ​​camera's tracking and shooting of the key target fails. It is necessary to compare the human feature data again in the captured image and re-select the third image.

[0182] In this embodiment, by randomly sampling fourth human feature data and matching it with the target feature queue, the AI ​​camera is able to continuously track the correct key target.

[0183] Optionally, the method for establishing a communication connection between the shooting gimbal and the mobile terminal also includes:

[0184] Obtain user operation commands on the mobile terminal;

[0185] Adjust the pitch and yaw angles of the shooting gimbal according to the operation instructions, and zoom in and out of the selected image in the operation instructions on the mobile terminal.

[0186] User commands include, but are not limited to, text input, voice input, and gesture control. If the user selects a specific point in the shooting frame to zoom in on the display screen, the gimbal automatically adjusts the shooting focus. If the user rotates the image by swiping left or right on the display screen, the gimbal automatically and uniformly adjusts the pitch and yaw angles to provide the user with the desired image.

[0187] In this embodiment, the pitch and yaw angles of the shooting gimbal are adjusted according to the user's operation commands, realizing real-time user interaction and improving the user experience.

[0188] Figure 3 is a schematic diagram of the interaction between the smart terminal and the camera gimbal according to an embodiment of this application, as shown in Figure 3:

[0189] Real-time communication between the mobile terminal and the shooting gimbal enables intelligent real-time tracking and shooting of key targets. The mobile terminal includes, but is not limited to, smartphones and cameras. The shooting gimbal integrates an NPU chip, a display screen, and an AI camera. The NPU chip integrates various machine learning models and algorithms, such as target detection models, face recognition and facial skeleton point detection models, face feature extraction models, human body feature extraction models, multi-target tracking algorithms, and position coordinate transformation algorithms, enabling real-time scene analysis and avoiding the burden and latency of cloud processing. The AI ​​camera uses a wide-angle lens and an AISoC (AI System on Chip) chip, which can be used for intelligent tracking and shooting of events, and can also be used to collect and integrate image data captured by the mobile terminal or another camera in the shooting gimbal. The display screen directly shows the execution results captured after the shooting gimbal receives drive commands, and it is aligned with the AI ​​camera, allowing users to directly control and interact with the shooting gimbal through the display screen.

[0190] In one possible implementation, to enable the shooting gimbal to interact directly with the user and correctly identify the user's operation commands, the AI ​​camera captures a specified motion of the user as the start of the operation command. This prevents unexpected interruptions or starts of shooting due to misrecognition of the user's actions or gestures during normal shooting and tracking scenarios. For example, the shooting gimbal displays the action command and a specified completion time on its screen. The action command includes, but is not limited to, opening a palm. The AI ​​camera only begins to recognize the user's operation command after recognizing that the user has completed the action command displayed on the screen within the specified time.

[0191] According to the embodiments of this disclosure, the following technical effects are achieved:

[0192] 1) It realizes real-time automated intelligent tracking control of AI cameras in shooting gimbals, which significantly improves the tracking and shooting efficiency of key targets.

[0193] 2) It integrates multi-dimensional perception data with various machine learning algorithms, ensuring the accuracy of key target positioning and high resolution of event shooting.

[0194] 3) A dynamic feedback adjustment mechanism has been established to ensure the effectiveness of AI cameras in intelligently tracking and capturing key targets through multiple authentications.

[0195] Example 2

[0196] This embodiment introduces a corresponding intelligent tracking shooting gimbal based on the intelligent tracking shooting method described in Embodiment 1. Figure 4 shows a block diagram of the intelligent tracking shooting gimbal device in this embodiment, and Figures 10 and 11 show three-dimensional structural schematic diagrams of the shooting gimbal; as shown in Figure 4, it includes:

[0197] An AI camera is used to acquire a first image containing the key target, input the first image into the target detection model to obtain the position of the key target; convert the position of the key target into first physical coordinates based on the distortion parameters and rotation angle of the AI ​​camera; perform intelligent tracking of the key target based on the first physical coordinates and a multi-target tracking algorithm; and send rotation commands to the drive mechanism.

[0198] Clamping structure for clamping and fixing camera device;

[0199] The drive mechanism is used to receive rotation commands and drive the shooting gimbal to rotate.

[0200] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the aforementioned method embodiment one, and will not be repeated here.

[0201] For a detailed description of the intelligent tracking shooting gimbal structure, please refer to Figures 10 and 11. This embodiment provides an intelligent tracking shooting gimbal that includes a gimbal base 1, a gimbal housing 2 with a receiving space, a tracker 3, and a drive mechanism for rotating the tracker 3. Both the tracker and the drive mechanism are disposed within the receiving space of the gimbal housing 2. The drive mechanism includes a first drive mechanism 41 and a second drive mechanism 42. The first drive mechanism 41 is driven and connected to the gimbal base 1 to achieve a 360-degree horizontal rotation of the tracker 3; the second drive mechanism 42 is driven and connected to the tracker 3 to achieve a 90-degree vertical tilt rotation of the tracker 3. Further, a clamping structure 6 for holding the camera device is fixedly disposed above the tracker 3; a switch button 7 is disposed on the gimbal housing 2.

[0202] The operating principle of the shooting gimbal in this embodiment is as follows: First, press the switch button 7 on the gimbal housing 2 to start the gimbal and wait for the tracker 3 to complete its reset self-test. Then, clamp the camera device, such as a smartphone, through the clamping structure 6 fixedly connected to the top of the tracker 3. Open the botgo application APP installed on the camera device and click "Add Device." The application will automatically connect to the shooting gimbal. During clamping, ensure that the camera of the camera device and the AI ​​camera of the shooting gimbal face the same direction. Under the action of the tracking information obtained by the AI ​​camera of the shooting gimbal, control the first drive mechanism 41 in combination with the second drive mechanism 42 to drive the AI ​​camera of the shooting gimbal to achieve 360-degree horizontal rotation and 90-degree vertical tilt rotation for all-round tracking. When the AI ​​camera tracks the movement, the camera of the shooting device can face different directions, that is, change the shooting angle of the camera of the shooting device, so as to shoot the target at different positions to capture static or dynamic images. This ensures that the camera of the shooting device is always aimed at the changing target, ensuring the accurate tracking function of the shooting gimbal. It has the beneficial effects of simple design structure and stable and convenient operation.

[0203] When the camera of the shooting device changes its shooting angle to follow the movement of the AI ​​camera, it effectively prevents the moving target from being outside the camera's capture range. This ensures the target remains within the camera's capture range, allowing the camera to capture static or dynamic images. For example, when the target is a dancer, the smartphone can effectively capture images of the dancer in motion. Similarly, when the target is a yoga practitioner, the smartphone can effectively capture images of the yoga practitioner in motion. And when the target is a soccer player, the smartphone can effectively capture dynamic images of the soccer ball and the player in motion. Therefore, by fixing the shooting device on the tracker 3 of the shooting gimbal, and controlling the movement of the tracker 3, the shooting angle of the camera is changed, ensuring the moving target remains within the camera's capture range, enabling the camera to capture the target effectively and accurately for both tracking and image acquisition.

[0204] Using the technical solution of this embodiment, the tracker 3 and the drive mechanism for driving the tracker 3 to rotate are both arranged in the receiving space of the gimbal housing. The tracker can be rotated horizontally 360 degrees by driving the first drive mechanism 41 connected to the gimbal base 1; and the tracker can be rotated vertically 90 degrees by driving the second drive mechanism 42 connected to the tracker 3. This achieves effective and accurate all-round tracking by the shooting gimbal, which has the advantages of simple structure, convenient use and high stability and accuracy of shooting gimbal control.

[0205] As a preferred embodiment, as shown in Figures 11 and 12, the tracker 3 of this embodiment includes an AI camera 31 and a tracking housing 32 for fixing the AI ​​camera 31. An arc-shaped rack 321 is provided on one side of the tracking housing 32. A drive gear adapted to the arc-shaped rack 321 is provided on the second drive mechanism 42. The drive motor of the second drive mechanism 42 drives the drive gear to rotate, which in turn drives the arc-shaped rack 321 to move, thereby realizing the precise and effective rotation of the tracker in the vertical direction.

[0206] It should be noted that the AI ​​camera 31 in this embodiment is the tracking sensor in application number 202423143844X.

[0207] Furthermore, as shown in Figure 11, a bearing seat 411 is provided in the middle part of the gimbal base 1. The driver of the first drive mechanism 41 is rotatably and fixedly connected to the bearing seat 411. The driver is a drive motor. The output end of the drive motor is fixedly connected to the bearing seat 411. Under the action of the AI ​​camera acquiring tracking target information, the drive motor of the first drive mechanism 41 is controlled to rotate, so as to realize the precise and effective rotation of the AI ​​camera in the horizontal direction.

[0208] As a preferred embodiment, as shown in FIG11, a control motherboard 9 is provided between the driver of the first drive mechanism 41 and the bearing seat 411. The control motherboard 9 is fixedly connected to the driver of the first drive mechanism 41, and both the control motherboard 9 and the driver of the first drive mechanism 41 are fixedly connected to the gimbal housing 2, which has the advantages of reasonable and compact structural design.

[0209] As a preferred embodiment, as shown in Figure 11, a support assembly is also provided within the accommodating space of the gimbal housing. The support assembly is fixedly connected to the gimbal housing. The support assembly includes a main support 51 and a secondary support 52. The secondary support 52 is disposed on both sides of the tracker and is provided with a bracket bearing. The tracking housing 32 of the tracker is provided with protruding columns 322 on both sides. The protruding columns 322 are rotatably and fixedly connected to the bracket bearings of the secondary support 52. The drive motor of the second drive mechanism 42 is fixed on the secondary support 52, thereby ensuring that the tracker can rotate accurately and effectively in the vertical direction.

[0210] Furthermore, as shown in Figure 11, the tracker 3 and the second drive mechanism 42 are disposed on one side of the main bracket 51, and a power supply 8 is disposed on the other side of the main bracket 51 to realize the power supply of the shooting gimbal.

[0211] Example 3

[0212] This embodiment is a preferred implementation based on Embodiment 2. The intelligent tracking and shooting gimbal in this embodiment is also suitable for intelligent tracking of jersey numbers. The specific tracking and shooting method can be found in the flowchart of the intelligent tracking and shooting method based on jersey numbers shown in Figure 5, which includes:

[0213] Obtain the first human image containing the key target.

[0214] The first human body image refers to a front and / or back photograph of a key target including their jersey number. The mobile terminal includes, but is not limited to, mobile phones, tablets, and cameras. This application does not restrict the method of obtaining the first human body image; it can be uploaded actively by the user, or it can be a partial view specified by the user in a real-time captured image through methods such as clicking or selecting a frame.

[0215] The first human body image is input into the target detection model, the number area in the first human body image is identified and cropped to obtain the first number image.

[0216] The target detection model is used to identify and locate the position information of the number region in the first human image from the image. In one embodiment, the target detection model includes an input layer, a backbone network layer, a Neck feature fusion layer, and a prediction layer. The input layer preprocesses the input image to meet the model's training and prediction requirements. The backbone network layer performs feature extraction, using a depth-optimized C2f module as the basic unit, which improves performance while reducing network size. The Neck feature fusion layer fuses feature maps from different stages of the backbone network layer to enhance feature representation capabilities. The prediction layer includes a SPPF (Spatial Pyramid Pooling Fast) module, a PAA (Probabilistic Anchor Assignment) module, a PAN (Path Aggregation Network) module, and a Head module. The SPPF module stitches together feature maps of different scales to improve the model's ability to detect targets of different sizes. The PAA module intelligently assigns anchor boxes to optimize the selection of positive and negative samples, improving the model's training effect. The PAN module performs path aggregation on features at different levels, enhancing the expressive power of the feature maps through bottom-up and top-down paths. The Head module is used for the final target detection and prediction, outputting the final position information of the number area in the first human body image, so as to accurately crop the number area.

[0217] Input the first jersey number image into the number classification model to obtain the first jersey number data.

[0218] The number classification model is used to identify and classify the numbers in the first number image.

[0219] Traditional jersey number recognition is typically achieved using OCR (Optical Character Recognition) technology. However, OCR technology performs poorly in dynamic images and fails to recognize accurate numbers when the jersey is wrinkled. Considering that the text involved in jersey number recognition may only contain the ten digits 0-9, using OCR technology would be wasteful of computing power and might also identify results other than the digits, further reducing accuracy. Therefore, in this embodiment, the jersey number recognition process is divided into two steps, S102 and S103, which sequentially perform number region recognition and number classification, thereby improving the accuracy of jersey number recognition.

[0220] In this application, there are no restrictions on the specific choice of number region identification and number classification models: the YOLOv8 (You Only Look Once version 8) model can be used to integrate number region identification and number classification; a combination of identification and classification models can also be used, such as convolutional neural networks, ViT (Vision Transformer), etc. for identification models; and large models such as MobileNet, EfficientNet-Lite or traditional machine learning models such as vector machines and random forests for classification models.

[0221] Based on the first jersey number data, key targets are matched and tracked in real-time shooting footage.

[0222] Figure 6 is a flowchart illustrating the intelligent tracking and shooting method based on jersey number according to an embodiment of this application, as shown in Figure 6:

[0223] First, a first human image containing the key target is acquired. The target detection model identifies the jersey number region in the first human image. If no jersey number region is found in the first human image, the jersey number recognition ends directly, the user is notified of recognition failure, and the user is prompted to re-upload or re-specify the first human image. If a jersey number region is found in the first human image, the jersey number region is cropped and then input into the number classification model to obtain the final first jersey number data, which serves as the basis for determining the key target.

[0224] To provide a more intuitive explanation, Figure 7 is a schematic diagram of jersey number recognition according to an embodiment of this application, as shown in Figure 7:

[0225] A two-stage image recognition algorithm was used to extract jersey number data. First, the number region in the image was identified and cropped using a target classification model. Then, the number classification model was used to identify the digits in the number region, outputting the number "17", which provides a basis for subsequent tracking and filming of player number 17.

[0226] In this embodiment, a target matching the first jersey number data is found and matched in the real-time shooting footage, i.e., the key target, and then tracked and shot to achieve intelligent tracking of the key target, thereby improving the user experience.

[0227] Optionally, based on the first jersey number data, key targets are matched and tracked in the live-stream footage, including:

[0228] The continuous real-time captured images are input into the target detection model, which sequentially identifies and crops the number regions in the real-time captured images to obtain the second number image queue.

[0229] The second jersey number data queue is obtained by sequentially classifying the second number images in the second number image queue using a number classification model.

[0230] The system searches for a second jersey number that matches the first jersey number in the second jersey number data queue. If a second jersey number that matches the first jersey number exists, the system tracks and captures the first human target corresponding to the second jersey number using a target tracking algorithm. If no second jersey number that matches the first jersey number exists, the system continues to identify and match.

[0231] For example, the first jersey number data is "7", and the second jersey number data queue in a live-action shot is ["1", "3", "5", "7"] (indicating that there are four players in the live-action shot at this time, with jersey numbers 1, 3, 5, and 7 respectively). A match of "7" is successfully found in the aforementioned queue. Therefore, the player with jersey number "7" in the live-action shot is identified as the key target, and a target tracking algorithm is used to track and capture the human target corresponding to jersey number "7" in the live-action shot.

[0232] Furthermore, during the tracking and filming of the first human target corresponding to the second jersey number data, a human feature extraction model can be used to extract the human features of the first human target, obtaining its human feature data. Then, the real-time captured footage is sequentially input into the human feature extraction model for feature extraction, obtaining human feature data from the real-time captured footage. From this human feature data, human targets similar to the first human target are selected for tracking and filming. The human feature extraction model includes, but is not limited to, CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), and LSTM (Long Short-Term Memory) networks, used to extract human feature data from the footage.

[0233] Preferably, since the sizes of multiple second number images may vary significantly, to aid model convergence, the second number images can be adjusted to a preset size, such as 64*64 pixels, before being input into the number classification model. Furthermore, during subsequent tracking and shooting, to address the issue of unclear images due to long shooting distances, the captured images can be segmented and enlarged before being input into the target detection model to adjust their resolution. These images are then input into the target detection and number classification models for recognition, and the recognition results are stitched together according to their original positions. This solves the problem of unclear recognition of numbers in human images due to their small size when shooting from a distance.

[0234] In this embodiment, real-time number recognition is performed on the live-shot footage to find the first human target whose jersey number matches the first jersey number data, ensuring the accuracy of number recognition and providing a foundation for subsequent tracking and shooting.

[0235] Optionally, a target tracking algorithm is used to track and capture images of the first human target in the second number frame, including:

[0236] The movement of the first human target is predicted by the target tracking algorithm to obtain the next predicted position of the first human target;

[0237] The camera automatically adjusts the pitch and yaw angles of the shooting gimbal based on the predicted position, continuously acquiring images of the second human body, including the first human target.

[0238] The specific choice of target tracking algorithm is not limited in this invention. Generally, mature target tracking algorithms such as MHT (Multiple Hypothesis Tracking), MOT (Multiple Object Tracking), and SORT (Simple Online and Realtime Tracking) can all implement the technical solution of this invention. In one possible implementation, the SORT algorithm is used for target tracking. SORT is a high-efficiency target tracking algorithm based on motion modeling and data association. Its core idea is to achieve continuous tracking of target identity across frames by fusing target motion prediction in the time dimension with detection box position information in the spatial dimension. First, Kalman filtering is used to predict the bounding box position of the first human target in the next frame based on its historical trajectory. Then, the Hungarian algorithm is introduced to establish the association between the current frame containing the first human target and the previous frame containing the first human target. By minimizing the association cost, the first human target in the current frame is matched with the first human target in the previous frame, ensuring that the bounding box of each first human target is associated with at most one predicted bounding box, thereby obtaining the next predicted position of the first human target.

[0239] In one possible implementation, the initial position of the first human target is (x1, y1), and the next predicted position of the first human target, as predicted by the target tracking algorithm, is (x2, y2). Then, the yaw angle Δφ that the gimbal needs to adjust is:

[0240] Where D represents the target horizontal parameter. The pitch angle Δθ that the shooting gimbal needs to adjust is:

[0241] In addition, the rotation speed of the shooting gimbal can be controlled by a PID controller. A smooth rotation speed command is generated based on the pitch angle difference or yaw angle difference to avoid gimbal shake. The formula for calculating the smooth rotation speed u(t) is as follows:

[0242] Where e(t) is the pitch angle difference or yaw angle difference, K p To set the hyperparameter to a value of 0.8, K i To set the hyperparameter to a value of 0.2, K d Set the hyperparameter to a value of 0.1.

[0243] In this embodiment, a target tracking algorithm is used to track and photograph the first human target, which improves the robustness in complex sports shooting scenarios.

[0244] Optionally, the method further includes:

[0245] The continuously acquired second human body images are sequentially identified and classified using object detection and number classification models to obtain a third jersey number data queue.

[0246] The third jersey number data in the third jersey number data queue is matched with the second jersey number data in turn. If the third jersey number data in the third jersey number data queue matches the second jersey number data, the tracking and shooting continues. If there is a third jersey number data in the third jersey number data queue that does not match the second jersey number data, the tracking and shooting fails, and the key target is matched again in the real-time shooting screen for tracking and shooting.

[0247] For example, if the second jersey number is "7", and the third jersey number data queue obtained by continuously identifying the second human body image through the target detection model and number classification model is ["7", "7", "7", "5"], then during the tracking and shooting process, if there is a third jersey number data in the third jersey number data queue that is inconsistent with the second jersey number data, the first human body target is lost and the tracking fails, then it is necessary to find and match the key target player with jersey number "7" again in the real-time shooting footage as the first human body target and restart the tracking and shooting, thereby avoiding mistakenly treating player number "5" as player number "7" and continuing to track and shoot.

[0248] In this embodiment, by matching the third jersey number data with the second jersey number data, the tracking target in complex scenes is effectively prevented from being incorrectly replaced. This is a real-time error correction mechanism that improves the accuracy and robustness of tracking and shooting.

[0249] Optionally, the continuous real-time captured images are input into the target detection model to sequentially identify and crop the number regions in the real-time captured images to obtain a second number image queue, and the method further includes:

[0250] When the live video is captured from a distance, the live video is divided into video blocks, and each video block is magnified by a preset magnification.

[0251] The number region is identified and cropped for each enlarged image block using an object detection model, resulting in a second number image queue.

[0252] The preset multipliers include, but are not limited to, 1x, 2x, 3x and 4x.

[0253] In this embodiment, the problem of unclear identification of numbers in human images due to small size is solved by segmenting and then magnifying the image.

[0254] Optionally, the method further includes:

[0255] If, during the tracking and filming process, the target detection model and number classification model identify and detect that the fourth jersey number data on the second human target in the real-time shooting image is consistent with the second jersey number data, then the tracking target will be transferred to the second human target.

[0256] For example, if the second jersey number is "7", and during the tracking and filming process, the target detection model and the number classification model identify that there is a fourth jersey number of "7" on the second human target in the real-time shooting footage, then there is a situation where the original tracking target is lost, resulting in tracking failure. It is necessary to transfer the tracking target to the second human target and start tracking and filming again, so as to avoid continuing to track and film the wrong tracking target as the "7" player.

[0257] In this embodiment, when the fourth jersey number data on the second human target is detected to be consistent with the second jersey number data, the tracking target is promptly transferred to ensure continuous tracking of the human target corresponding to the number, rather than losing the target or mistakenly tracking an irrelevant person.

[0258] Optionally, the method further includes:

[0259] The first human image and the first jersey number data are added to the training data of the object detection model and the number classification model on the server side for model iteration.

[0260] Figure 8 is a schematic diagram of the offline training process of the model according to an embodiment of this application, as shown in Figure 8:

[0261] The real-time inference model (i.e., the object detection model and the number classification model) is deployed offline for training on the server side. The training data includes, but is not limited to, basketball, football, volleyball, and rugby matches. When performing jersey number recognition, the pre-trained real-time inference model (i.e., the object detection model and the number classification model) is pulled to the mobile terminal for execution, thereby achieving efficient real-time jersey number recognition. During this process, the first human image and the first jersey number after recognition are collected and fed back to the server side as new training data to supplement the model's offline training set. Through this data feedback mechanism, the model can continuously learn new scene features and recognition patterns, thereby iteratively optimizing.

[0262] In this embodiment, by supplementing the first human image and the first jersey number data into the offline data on the server, the generalization ability and recognition accuracy of the model are continuously improved, and the model performance is optimized.

[0263] According to the embodiments of this application, the following technical effects are achieved:

[0264] 1) Real-time and accurate identification of jersey numbers was achieved through target detection and number classification models, providing a foundation for subsequent match tracking and filming.

[0265] 2) When shooting from a distance, the method of segmentation, magnification and parallel reasoning recognition effectively solves the problem of inaccurate recognition of jersey numbers when shooting from a distance due to their small size.

[0266] 3) After identifying the jersey numbers, the identification resources from online inference were used as the data source for model iteration, thus constructing an efficient data feedback mechanism.

[0267] Example 4

[0268] This embodiment uses the device embodiment corresponding to the intelligent tracking and shooting method based on jersey number described in Embodiment 3 to further illustrate the solution described in this application.

[0269] Figure 9 shows a block diagram of a jersey number-based intelligent tracking and shooting platform according to an embodiment of this application, as shown in Figure 9, including:

[0270] Acquisition module 501 is used to acquire a first human image containing key targets;

[0271] The recognition module 502 is used to input the first human body image into the target detection model, recognize the number area in the first human body image and crop it to obtain the first number image;

[0272] The classification module 503 is used to input the first number image into the number classification model to obtain the first jersey number data;

[0273] The tracking and shooting module 504 is used to match key targets in the real-time shooting screen based on the first jersey number data and to perform tracking and shooting.

[0274] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the aforementioned method embodiment three, and will not be repeated here.

[0275] Figure 13 shows a schematic diagram of a terminal device or server suitable for implementing embodiments of this application.

[0276] As shown in Figure 13, the terminal device or server includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the terminal device or server. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0277] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0278] Specifically, according to embodiments of this application, the above method flow steps can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined in the system of this application.

[0279] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0280] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0281] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor. The names of these units or modules do not, in certain circumstances, constitute a limitation on the unit or module itself.

[0282] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium stores one or more programs that, when used by one or more processors, execute the methods described in this application.

[0283] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing application concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions claimed in this application.

Claims

1. A smart tracking and shooting gimbal, characterized in that, It includes a gimbal base, a gimbal housing, a tracker, and a drive mechanism, wherein the tracker and drive mechanism are disposed within the gimbal housing; The tracker includes an AI camera, which is used to acquire a first image containing a key target, input the first image into a target detection model to obtain the position of the key target; convert the position of the key target into first physical coordinates according to the distortion parameters and rotation angle of the AI ​​camera; intelligently track the key target according to the first physical coordinates and a multi-target tracking algorithm, and send a rotation command to the drive mechanism. The drive mechanism is used to receive the rotation command and drive the shooting gimbal to rotate; the drive mechanism includes a first drive mechanism and a second drive mechanism, the first drive mechanism is driven and connected to the gimbal base to realize the horizontal 360-degree rotation of the tracker; the second drive mechanism is driven and connected to the tracker to realize the vertical 90-degree tilt rotation of the tracker.

2. The intelligent tracking and shooting gimbal according to claim 1, characterized in that, The intelligent tracking and imaging of the key target based on the first physical coordinates and a multi-target tracking algorithm includes: The multi-target tracking algorithm predicts the movement of the key target based on the first physical coordinates and obtains the second physical coordinates of the key target. The camera automatically adjusts the pitch and yaw angles of the shooting gimbal based on the second physical coordinates to continuously acquire a second image containing the key target.

3. The intelligent tracking and shooting gimbal according to claim 2, characterized in that, It also includes face recognition and face skeleton detection models, face feature extraction models, and human body feature extraction models; The face recognition and face skeleton point detection model and the face feature extraction model are used to extract face features from the first image to obtain the first face feature data; The human body feature extraction model is used to extract human body features from the first image to obtain first human body feature data. The second image is sequentially input into the human feature extraction model for feature extraction to obtain second human feature data. Third human feature data similar to the first human feature data is selected from the second human feature data, and intelligent target tracking and shooting are performed on the third image corresponding to the third human feature data. The face recognition and face skeleton point detection model and the face feature extraction model are used to extract face features from the third image to obtain second face feature data. The third image is then reconfirmed based on the first face feature data and the second face feature data.

4. The intelligent tracking and shooting gimbal according to claim 3, characterized in that, The step of extracting facial features from the first image using a face recognition and facial skeleton point detection model and a face feature extraction model to obtain first facial feature data includes: The face recognition and face skeleton point detection model is used to locate and align the face in the first image to obtain the first face data and the first face skeleton point data. The first face data and the first face skeleton point data are input into the face feature extraction model to obtain the first face feature data.

5. The intelligent tracking and shooting gimbal according to claim 3, characterized in that, The step of performing secondary confirmation of the third image based on the first facial feature data and the second facial feature data includes: The first facial feature data and the second facial feature data are matched for similarity. If the first facial feature data and the second facial feature data are similar, then the intelligent target tracking and shooting will continue. If the first facial feature data and the second facial feature data are not similar, then the third image is re-filtered.

6. The intelligent tracking and shooting gimbal according to claim 3, characterized in that, When performing intelligent target tracking and shooting on the key target, the third frame is randomly selected and input into the human body feature extraction model for feature extraction to obtain the fourth human body feature data. The fourth human body feature data is matched with the target feature queue for similarity matching; If the fourth human feature data is similar to the target feature queue, then the intelligent target tracking and shooting will continue; If the fourth human feature data is not similar to the target feature queue, the intelligent target tracking shooting fails, the intelligent target tracking shooting is paused, and the third screen is selected again; The target feature queue is a set of feature sequences of all historically collected third-party human feature data.

7. The intelligent tracking and shooting gimbal according to claim 3, characterized in that, The camera gimbal is communicatively connected to the mobile terminal, specifically: Obtain user operation commands on the mobile terminal; Adjust the pitch and yaw angles of the shooting gimbal according to the operation instructions, and zoom in and out on the screen selected by the operation instructions in the mobile terminal.

8. The intelligent tracking and shooting gimbal according to claim 1, characterized in that, A clamping structure for holding the camera device is fixedly installed above the tracker.

9. The intelligent tracking and shooting gimbal according to claim 1, characterized in that, The tracker also includes a tracking housing for fixing the AI ​​camera, and an arc-shaped rack is provided on one side of the tracking housing. The second drive mechanism is provided with a drive gear that is adapted to the arc-shaped rack.

10. The intelligent tracking and shooting gimbal according to claim 1, characterized in that, The gimbal base has a bearing seat in the middle part, and the driver of the first drive mechanism is rotatably connected to the bearing seat.

11. The intelligent tracking and shooting gimbal according to claim 10, characterized in that, A control motherboard is provided between the driver of the first drive mechanism and the bearing seat, and the control motherboard and the driver of the first drive mechanism are fixedly connected to the gimbal housing.

12. The intelligent tracking and shooting gimbal according to claim 1, characterized in that, The gimbal housing also includes a support assembly, which is fixedly connected to the gimbal housing. The support assembly includes a main support and an auxiliary support, with the auxiliary support located on both sides of the tracker.

13. The intelligent tracking and shooting gimbal according to claim 12, characterized in that, The bracket is provided with a support bearing, and the tracker has protruding columns on both sides, which are rotatably connected to the support bearing.

14. The intelligent tracking and shooting gimbal according to claim 1, characterized in that, The AI ​​camera is also suitable for intelligent tracking and capturing jersey numbers, specifically including: Acquire the first human image containing the key target; The first human body image is input into the target detection model, the number area in the first human body image is identified and cropped to obtain the first number image; Input the first number image into the number classification model to obtain the first jersey number data; The key target is matched and tracked in real-time shooting footage based on the first jersey number data.

15. The intelligent tracking and shooting gimbal according to claim 14, characterized in that, The step of matching the key target in the real-time shooting footage based on the first jersey number data and tracking and shooting it includes: The continuous real-time captured images are input into the target detection model, and the number regions in the real-time captured images are identified and cropped in sequence to obtain the second number image queue; The second jersey number data queue is obtained by sequentially classifying the second number images in the second number image queue using a number classification model. Search for a second jersey number that matches the first jersey number in the second jersey number data queue. If a second jersey number that matches the first jersey number exists, track and photograph the first human target corresponding to the second jersey number using a target tracking algorithm. If no second jersey number that matches the first jersey number exists, continue the identification and matching process.

16. The intelligent tracking and shooting gimbal according to claim 15, characterized in that, The step of tracking and capturing the first human target corresponding to the second jersey number data using a target tracking algorithm includes: The target tracking algorithm is used to predict the movement of the first human target and obtain the next predicted position of the first human target. The camera automatically adjusts the pitch and yaw angles of the shooting gimbal based on the predicted position for the next step, and continuously acquires images of a second human body containing the first human target.

17. The intelligent tracking and shooting gimbal according to claim 16, characterized in that, The target detection model and the number classification model are used to sequentially identify and classify the continuously acquired second human body images to obtain a third jersey number data queue. The third jersey number data in the third jersey number data queue is matched with the second jersey number data in turn. If the third jersey number data in the third jersey number data queue and the second jersey number data are both consistent, the tracking and shooting continues. If there is a third jersey number data in the third jersey number data queue that is inconsistent with the second jersey number data, the tracking and shooting fails, and the key target is matched again in the real-time shooting screen for the tracking and shooting.

18. The intelligent tracking and shooting gimbal according to claim 15, characterized in that, The step of inputting continuous real-time captured images into the target detection model, sequentially identifying and cropping the number regions in the real-time captured images, and obtaining a second number image queue, further includes: When the real-time shooting footage is captured from a distance, the real-time shooting footage is divided into image blocks, and each image block is magnified by a preset magnification. The target detection model is used to identify and crop the number region for each magnified image block to obtain the second number image queue.

19. The intelligent tracking and shooting gimbal according to claim 17, characterized in that, If, during the tracking and shooting process, the target detection model and the number classification model identify that the fourth jersey number data on the second human target in the real-time shooting image is consistent with the second jersey number data, then the tracking target of the tracking and shooting is transferred to the second human target.

20. The intelligent tracking and shooting gimbal according to claim 14, characterized in that, Also includes: The first human image and the first jersey number data are added to the training data of the target detection model and the number classification model on the server side for model iteration.

21. An intelligent tracking and shooting method, characterized in that, include: The first image containing the key target is captured by the AI ​​camera in the shooting gimbal. The first image is then input into the target detection model to obtain the location of the key target. The key target position is converted into first physical coordinates based on the distortion parameters and rotation angle of the AI ​​camera; The key target is intelligently tracked and photographed based on the first physical coordinates and a multi-target tracking algorithm.

22. The intelligent tracking and shooting method according to claim 21, characterized in that, The intelligent tracking and imaging of the key target based on the first physical coordinates and a multi-target tracking algorithm includes: The multi-target tracking algorithm predicts the movement of the key target based on the first physical coordinates and obtains the second physical coordinates of the key target. The camera automatically adjusts the pitch and yaw angles of the shooting gimbal based on the second physical coordinates to continuously acquire a second image containing the key target.

23. The intelligent tracking and shooting method according to claim 22, characterized in that, The method further includes: The face features of the first image are extracted using a face recognition and face skeleton point detection model and a face feature extraction model to obtain the first face feature data. Human features are extracted from the first image using a human feature extraction model to obtain first human feature data. The second image is sequentially input into the human feature extraction model for feature extraction to obtain second human feature data. Third human feature data similar to the first human feature data is selected from the second human feature data, and intelligent target tracking and shooting are performed on the third image corresponding to the third human feature data. The face recognition and face skeleton point detection model and the face feature extraction model are used to extract face features from the third image to obtain second face feature data. The third image is then reconfirmed based on the first face feature data and the second face feature data.

24. The intelligent tracking and shooting method according to claim 23, characterized in that, The step of extracting facial features from the first image using a face recognition and facial skeleton point detection model and a face feature extraction model to obtain first facial feature data includes: The face recognition and face skeleton point detection model is used to locate and align the face in the first image to obtain the first face data and the first face skeleton point data. The first face data and the first face skeleton point data are input into the face feature extraction model to obtain the first face feature data.

25. The intelligent tracking and shooting method according to claim 23, characterized in that, The step of performing secondary confirmation of the third image based on the first facial feature data and the second facial feature data includes: The first facial feature data and the second facial feature data are matched for similarity. If the first facial feature data and the second facial feature data are similar, then the intelligent target tracking and shooting will continue. If the first facial feature data and the second facial feature data are not similar, then the third image is re-filtered.

26. The intelligent tracking and shooting method according to claim 23, characterized in that, The method further includes: When performing intelligent target tracking and shooting on the key target, the third frame is randomly selected and input into the human body feature extraction model for feature extraction to obtain the fourth human body feature data. The fourth human body feature data is matched with the target feature queue for similarity matching; If the fourth human feature data is similar to the target feature queue, then the intelligent target tracking and shooting will continue; If the fourth human feature data is not similar to the target feature queue, the intelligent target tracking shooting fails, the intelligent target tracking shooting is paused, and the third screen is selected again; The target feature queue is a set of feature sequences of all historically collected third-party human feature data.

27. The intelligent tracking and shooting method according to claim 23, characterized in that, The shooting gimbal is communicatively connected to the mobile terminal, and the method further includes: Obtain user operation commands on the mobile terminal; Adjust the pitch and yaw angles of the shooting gimbal according to the operation instructions, and zoom in and out on the screen selected by the operation instructions in the mobile terminal.

28. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 21-27.

29. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 21-27.

Images