Motion capture system and motion capture method

Abstract

A motion capture system and a motion capture method, relating to the technical field of motion capture. The motion capture system comprises: a motion capture module, which comprises sensors of multiple modalities and is used for sensing a measured target to obtain sensing data of different modalities; and a processing unit, which is connected to the motion capture module and is used for processing the sensing data of different modalities to obtain motion data of the measured target.

Description

Motion capture system and motion capture method

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese patent applications filed on January 9, 2025, No. 202510033674.X, No. 202510033677.3, and No. 202510033678.8, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of motion capture technology, and more specifically, to a motion capture system and a motion capture method. Background Technology

[0004] Optical motion capture technology is widely used in many fields due to its high precision and fast response. This technology mainly relies on near-infrared motion capture cameras to monitor and track specific markers installed on the target object. Multiple near-infrared cameras are arranged around the target object, and the marker is captured by two or more cameras at the same time, which can determine the three-dimensional spatial position of the point. Through continuous shooting, the movement trajectory of the marker over time can be tracked. However, in complex environments, there are certain limitations to sensing the target object by relying solely on near-infrared cameras. Summary of the Invention

[0005] This application proposes a motion capture system and method that improves the accuracy and robustness of motion capture.

[0006] In a first aspect, this application provides a motion capture system, the system comprising:

[0007] The motion capture module includes sensors of various modalities, used to sense the target under test and obtain sensing data of different modalities;

[0008] The processing unit, connected to the motion capture module, is used to process the sensing data of the different modalities to obtain the motion data of the target being measured.

[0009] Secondly, this application provides a motion capture method, the method comprising:

[0010] The target under test is perceived to obtain perception data in different modalities;

[0011] The sensing data of the different modalities are processed to obtain the motion data of the target being measured.

[0012] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the motion capture method as described in the second aspect above.

[0013] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the motion capture method as described in the second aspect above.

[0014] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the motion capture method as described in the second aspect.

[0015] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the motion capture method as described in the second aspect above.

[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0017] Figure 1 is a schematic diagram of one of the motion capture systems provided in some embodiments of this application;

[0018] Figure 2 is a second schematic diagram of the structure of a motion capture system provided in some embodiments of this application;

[0019] Figure 3 is a third schematic diagram of the structure of a motion capture system provided in some embodiments of this application;

[0020] Figure 4 is a fourth schematic diagram of the structure of a motion capture system provided in some embodiments of this application;

[0021] Figure 5 is a schematic diagram of the structure of a motion capture system provided in some embodiments of this application;

[0022] Figure 6 is a schematic diagram of the structure of a motion capture system provided in some embodiments of this application;

[0023] Figure 7 is a schematic diagram of the structure of a motion capture system provided in some embodiments of this application;

[0024] Figure 8 is a schematic diagram of the structure of a motion capture system provided in some embodiments of this application;

[0025] Figure 9 is a schematic diagram of the structure of a motion capture system provided in some embodiments of this application;

[0026] Figure 10 is a schematic diagram of the structure of a motion capture system provided in some embodiments of this application;

[0027] Figure 11 is a flowchart illustrating a motion capture method provided in some embodiments of this application;

[0028] Figure 12 is a schematic diagram of the structure of an electronic device provided in some embodiments of this application.

[0029] Explanation of reference numerals in the attached figures: 101: Motion capture module; 102: Processing unit; 103: Non-visible light image sensor; 104: Visible light image sensor; 105: Complementary light source; 106: First image processing unit; 107: Second image processing unit; 108: Inertial sensor; 1200: Electronic device; 1201: Processor; 1202: Memory. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained based on the embodiments of this application are within the scope of protection of this application.

[0031] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0032] The motion capture system and motion capture method provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0033] Figure 1 is a schematic diagram of one of the embodiments of the motion capture system provided in this application. As shown in Figure 1, the motion capture system includes:

[0034] The motion capture module 101 includes sensors of multiple modalities for sensing the target under test and obtaining sensing data of different modalities;

[0035] The processing unit 102 is connected to the motion capture module 101 and is used to process the sensing data of different modalities to obtain the motion data of the target being measured.

[0036] It should be noted that the target being tested in the embodiments of this application may be one or more targets, and this application does not impose any restrictions on this.

[0037] The motion capture module 101 includes sensors of various modalities, which can be of different types, such as non-visible light image sensors and visible light image sensors. Each modal sensor can sense the target and obtain specific modal sensing data; for example, a near-infrared image sensor senses the target and obtains a near-infrared image, while a visible light image sensor senses the target and obtains a visible light image. Through these multiple modal sensors, each motion capture module 101 can sense the target from multiple modalities, improving the comprehensiveness and accuracy of target sensing. In some embodiments, the Zhang Zhengyou calibration method is used to calibrate the sensors of each modality.

[0038] In some embodiments, the motion capture system may include multiple motion capture modules 101. When the line of sight of one motion capture module 101 is blocked or malfunctions, other motion capture modules 101 can continue to perceive the target under test, thereby improving the robustness of the motion capture system. Multiple motion capture modules can perceive the target under test from different angles (multiple viewpoints), thereby improving the comprehensiveness and accuracy of the perception of the target under test. The collaborative work of multiple motion capture modules can improve the accuracy and robustness of motion capture.

[0039] In some embodiments, the motion capture module 101 includes at least one non-visible light image sensor and at least one visible light image sensor for sensing the target under test, obtaining non-visible light images and visible light images of the target under test respectively, thereby realizing the sensing of the target under test from both optical and visual dual modalities.

[0040] The processing unit 102 is used to process sensing data of different modalities, such as data matching, fusion, and analysis, to obtain the motion data of the target under test. The motion data of the target under test includes the position data and attitude data of the target under test in three-dimensional space, that is, pose data, thereby realizing the motion capture of the target under test.

[0041] Motion data is obtained by combining sensing data from different modalities, which realizes the fusion of sensing data from different modalities and improves the accuracy of motion capture system in capturing the target. Single-modal sensors may be affected by occlusion or other factors, resulting in inaccurate output sensing data. By fusing sensing data from multiple modal sensors, the robustness of motion capture system can be improved.

[0042] In the above technical solution, the motion capture system includes motion capture modules, each of which includes sensors of multiple modalities. The sensors are used to sense the target being measured and obtain sensing data of different modalities. The processing unit is connected to the motion capture module and is used to process the sensing data of different modalities to obtain the motion data of the target being measured. This realizes the motion capture of the target being measured. By obtaining motion data based on sensing data of different modalities, the accuracy and robustness of the motion capture system in capturing the target being measured are improved.

[0043] Figure 2 is a second schematic diagram of the structure of a motion capture system provided in some embodiments of this application. As shown in Figure 2, in some embodiments, the motion capture module 101 includes at least one non-visible light image sensor 103 and at least one visible light image sensor 104.

[0044] The non-visible light image sensor 103 is used to sense the target under test in the near-infrared or infrared band and output an image in the near-infrared or infrared band.

[0045] The visible light image sensor 104 is used to sense the target under test in the visible light band and output a visible light image.

[0046] In some embodiments, the non-visible light image sensor 103 is used to sense the target under test in the near-infrared or infrared band and output images in the near-infrared or infrared band. Since the light in the near-infrared and infrared bands is invisible to the human eye, the non-visible light image sensor is not limited by visible light conditions and is more suitable for environments with less than ideal visible light conditions. The motion capture module includes at least one non-visible light image sensor 103, which improves the applicability of the motion capture system under different lighting conditions. The visible light image sensor 104 is used to sense the target under test in the visible light band and output visible light images. Under good lighting conditions, the visible light image sensor 104 can output high-resolution color images.

[0047] The motion capture module 101 includes at least one non-visible light image sensor 103 and at least one visible light image sensor 104, enabling the motion capture module to perceive the target in both optical and visual modes. The optical mode corresponds to the near-infrared or infrared image output by the non-visible light image sensor 103, while the visual mode corresponds to the visible light image output by the visible light image sensor 104. This improves the comprehensiveness and accuracy of the perception of the target. Motion data is obtained based on the perception data of different modes, which also improves the accuracy of the motion capture system in capturing the motion of the target.

[0048] In the above technical solution, the motion capture module includes at least one non-visible light image sensor and at least one visible light image sensor. The non-visible light image sensor is used to sense the target under test in the near-infrared or infrared band and output an image in the near-infrared or infrared band. The visible light image sensor is used to sense the target under test in the visible light band and output a visible light image. This realizes optical and visual dual-modal perception of the target under test and improves the accuracy and robustness of the motion capture system in motion capture of the target under test.

[0049] Figure 3 is a third schematic diagram of the structure of a motion capture system provided in some embodiments of this application. As shown in Figure 3, in some embodiments, the motion capture module 101 further includes at least one near-infrared or infrared supplementary light source 105 for illuminating the target being measured;

[0050] At least one marker is arranged on the target being measured. Under the illumination of the supplementary light source, the marker has a contrast with the background in the imaging of the non-visible light image sensor 103.

[0051] The near-infrared or infrared band supplementary light source 105 is used to provide near-infrared or infrared band light to illuminate the target under test, so that the non-visible light image sensor 103 can perceive the target under test more clearly, improve the quality of the near-infrared or infrared band image output by the non-visible light image sensor 103, and also enable the non-visible light image sensor 103 to still perceive the target under near-infrared or infrared band lighting conditions that are not ideal.

[0052] At least one marker is placed on the target being measured. The marker is a specially designed mark or luminous point, also known as a marker. Under the illumination of the supplementary light source 105, the marker will create a contrast with the background in the imaging of the non-visible light image sensor, making the marker stand out more in the near-infrared or infrared band image output by the non-visible light image sensor 103. This contrast with the surrounding background makes the marker easier to identify and track in subsequent image processing and analysis, improving the accuracy of marker identification and positioning.

[0053] In the above technical solution, the motion capture module includes at least one near-infrared or infrared supplementary light source for illuminating the target under test, and at least one marker point is arranged on the target under test. Under the illumination of the supplementary light source, the marker point has a contrast with the background in the imaging of the non-visible light image sensor, making the marker point more prominent, improving the accuracy of identifying and locating the marker point in the imaging of the non-visible light image sensor, and thus improving the accuracy of the motion capture module in motion capture of the target under test.

[0054] As shown in Figure 3, in some embodiments, the processing unit 102 is specifically used for:

[0055] The coordinates of the marker points are obtained by tracking and spatially locating the marker points in near-infrared or infrared images.

[0056] Keypoint detection is performed on visible light images to obtain keypoint information, which includes the coordinates and ID of the keypoints.

[0057] Motion attitude estimation of the target in the visible light image is performed to obtain the first pose data;

[0058] Match the coordinates of key points with the coordinates of marker points to determine the ID of the marker points;

[0059] Based on the coordinates and ID of the marker, the marker is reconstructed in three dimensions to obtain its three-dimensional coordinates;

[0060] Based on the three-dimensional coordinates of at least three non-collinear marker points, the pose of the target under test is calculated to obtain the second pose data;

[0061] The motion data of the target object are obtained by fusing the first pose data and the second pose data.

[0062] In some embodiments, at least one marker is arranged on the target under test. Under the illumination of the supplementary light source, these markers contrast with the background in the near-infrared or infrared band image output by the non-visible light image sensor, appearing as bright spots. Through image processing technology, the processing unit 102 can track the markers in the near-infrared or infrared band image and obtain the coordinates of the markers through spatial positioning calculation.

[0063] In some embodiments, the processing unit 102 employs deep learning algorithms, such as YOLOv8, to detect key points in the visible light image, obtaining the coordinates and IDs of the key points, where the ID is a unique identifier for the key point. In some embodiments, the processing unit 102 employs algorithms such as MediaPipe to estimate the motion pose of the target in the visible light image, obtaining first pose data. The pose data includes the position and pose data of the target in three-dimensional space.

[0064] The processing unit 102 matches the coordinates of the marker points with the coordinates of the key points, establishes a correspondence between the marker points and the key points, and assigns an ID to each marker point. In some embodiments, assigning an ID to a marker point includes: comparing the coordinates of the key points and the coordinates of the marker points, determining the correspondence between the marker points and the key points by finding the nearest match, assigning a consistent ID value to marker points that match key points in the same spatial location, so that each marker point has a unique and temporally consistent ID.

[0065] The processing unit 102 uses triangulation to reconstruct the three-dimensional coordinates of the marker points based on their coordinates and IDs. Based on the three-dimensional coordinates of at least three non-collinear marker points, the processing unit 102 can perform pose calculation on the target to obtain the second pose data.

[0066] The first pose data is determined based on a visible light image, while the second pose data is determined based on an image in the near-infrared or infrared bands. In some embodiments, fusion algorithms such as Kalman filtering are used to fuse the first and second pose data to obtain the motion data of the target. This achieves the fusion of pose data obtained from different modal sensing data. The fusion calculation can effectively improve the anti-occlusion capability of the target motion solution while ensuring controllable accuracy loss, thereby improving the accuracy and robustness of the motion capture system in capturing the target.

[0067] In some embodiments, the motion capture system includes at least one motion capture module 101, which can use near-infrared or infrared images and visible light images obtained from sensing the object under test from at least one viewpoint to perform three-dimensional reconstruction of the object under test using triangulation, obtain the three-dimensional coordinates of the marker points, and calculate the pose of the three-dimensional coordinates of at least three non-collinear marker points, thereby improving the accuracy and robustness of the motion capture system in motion capture of the target under test.

[0068] In the above technical solution, the processing unit is used to track and spatially locate marker points in near-infrared or infrared images to obtain marker point coordinates, detect key points in visible light images to obtain key point coordinates and IDs, estimate the motion posture of the target in the visible light image to obtain first pose data, match the coordinates of key points and marker points to determine the ID of the marker points, perform 3D reconstruction of the marker points based on the coordinates and IDs of the marker points to obtain the 3D coordinates of the marker points, perform pose calculation on the target based on the 3D coordinates of at least three non-collinear marker points to obtain second pose data, and fuse the first pose data and second pose data to obtain the motion data of the target. This realizes the fusion of pose data obtained from different modal sensing data, improving the accuracy and robustness of the motion capture system for motion capture of the target.

[0069] Figure 4 is a fourth schematic diagram of the structure of a motion capture system provided in some embodiments of this application. As shown in Figure 4, in some embodiments, the motion capture system further includes: a first image processing unit 106, connected to a non-visible light image sensor 103 and a visible light image sensor 104.

[0070] The first image processing unit 106 is used for:

[0071] The system tracks and spatially locates markers in near-infrared or infrared images, and sends the coordinates of the markers to the processing unit 102.

[0072] In some embodiments, the motion capture module 101 includes a non-visible light image sensor 103 and a visible light image sensor 104 for sensing the target under test and obtaining near-infrared or infrared images and visible light images. The processing unit 102 processes the near-infrared or infrared images and visible light images to obtain motion data of the target under test and realize motion capture of the target under test.

[0073] In some embodiments, the motion capture module 101 may also include a first image processing unit 106, which uses image processing technology to track marker points in near-infrared or infrared band images and calculates the coordinates of the marker points through spatial positioning. The processing unit 102 receives the coordinates of the marker points output by the first image processing unit 106, thereby improving the flexibility of the motion capture system.

[0074] In some embodiments, the first image processing unit 106 may also be an embedded image processing unit, such as a field programmable gate array (FPGA), embedded in the motion capture module 101 and connected to the non-visible light image sensor 103.

[0075] In the above technical solution, the motion capture module also includes a first image processing unit, which is connected to a non-visible light image sensor and a visible light image sensor. It is used to track and spatially locate markers in near-infrared or infrared images and output the coordinates of the markers to the processing unit, thereby improving the flexibility of the motion capture system.

[0076] In some embodiments, the first image processing unit 106 is also connected to the visible light image sensor 104 and is further used for:

[0077] The visible light image is subjected to one or more of the following processes: feature extraction, feature recognition, feature spatial positioning calculation, motion data solution, and the processing results are sent to the processing unit 102.

[0078] In some embodiments, feature extraction refers to extracting features from a visible light image that can be used for subsequent motion capture of the target under test. Features may include edges, corners, textures, etc. in the image. Feature recognition refers to the process of determining the specific identity or category of these features based on feature extraction. Feature spatial localization calculation refers to estimating the spatial position and pose of the target under test by recognizing and using features in the image. Motion data solving refers to the process of determining the motion data of the target under test.

[0079] In the above technical solution, the first image processing unit is also used to perform feature extraction, feature recognition, feature space positioning calculation or motion data calculation on the visible image, and send the processing results to the processing unit, thereby improving the flexibility of the motion capture system.

[0080] In some embodiments, the first image processing unit 106 is specifically used for:

[0081] Keypoint detection is performed on visible light images to obtain keypoint information, which includes the coordinates and ID of the keypoints.

[0082] Motion attitude estimation of the target in the visible light image is performed to obtain the first pose data;

[0083] Match the coordinates of key points with the coordinates of marker points to determine the ID of the marker points;

[0084] The coordinates of the marker, the ID of the marker, and the first pose data are sent to the processing unit 102.

[0085] In some embodiments, the first image processing unit 106 employs deep learning algorithms, such as YOLOv8, to perform keypoint detection on the visible light image, obtaining the coordinates and IDs of the keypoints, where the ID is a unique identifier for the keypoint. In some embodiments, the first image processing unit 106 uses algorithms such as MediaPipe to estimate the motion pose of the target in the visible light image, obtaining first pose data.

[0086] The first image processing unit 106 is further configured to match the coordinates of the marker points with the coordinates of the key points, establish a correspondence between the marker points and the key points, and assign a unique ID to each marker point. In some embodiments, assigning an ID to a marker point includes: comparing the coordinates of the key points and the coordinates of the marker points, determining the correspondence between the marker points and the key points by finding the nearest match, assigning a consistent ID value to the marker points that match the key points at the same spatial location, so that the marker points have unique and temporally consistent IDs.

[0087] In the above technical solution, the first image processing unit is used to detect key points in the visible light image, obtain the coordinates and IDs of the key points, estimate the motion posture of the target in the visible light image, obtain the first pose data, match the coordinates of the key points and the coordinates of the marker points, determine the IDs of the marker points, and send the coordinates of the marker points, the IDs of the marker points, and the first pose data to the processing unit. This realizes the data processing of the visible light image in the first image processing unit, which improves the flexibility of the motion capture system.

[0088] In some embodiments, the processing unit 102 is used to:

[0089] Based on the coordinates and ID of the marker, the marker is reconstructed in three dimensions to obtain its three-dimensional coordinates;

[0090] Based on the three-dimensional coordinates of at least three non-collinear marker points, the pose of the target under test is calculated to obtain the second pose data;

[0091] The motion data of the target object are obtained by fusing the first pose data and the second pose data.

[0092] After receiving the coordinates of the marker point, the ID of the marker point, and the first pose data sent by the first image processing unit 106, the processing unit 102 performs three-dimensional reconstruction of the marker point using triangulation based on the coordinates and ID of the marker point to obtain the three-dimensional coordinates of the marker point; based on the three-dimensional coordinates of at least three non-collinear marker points, the processing unit 102 can perform pose calculation on the target under test to obtain the second pose data.

[0093] The second pose data of the target is obtained by calculating the pose of the target based on the 3D coordinates of at least three non-collinear marker points. For example, an object coordinate system can be constructed using the three marker points, with one marker point as the origin, the line connecting the origin and another marker point as the x-axis, and the plane defined by the three marker points as the xy-plane. The z-axis direction is then determined through vector cross product, and the rotation matrix and translation vector are derived to obtain the position and pose data of the target. Alternatively, the second pose data of the target can be calculated using the known 3D coordinates of three or more non-collinear marker points and their 2D projection coordinates in the image. In some embodiments, features of the target, such as joint topology or continuous motion constraints, can be combined to optimize the pose calculation process and improve the accuracy and reliability of the pose calculation.

[0094] In some embodiments, the processing unit 102 uses fusion algorithms such as Kalman filtering to fuse the first pose data and the second pose data to obtain the motion data of the target under test. This realizes the fusion of motion data obtained from perception data based on different modalities. The fusion calculation can effectively improve the anti-occlusion of the target motion solution while ensuring that the accuracy loss is controllable, and improve the accuracy and robustness of the motion capture system in motion capture of the target under test.

[0095] In the above technical solution, the processing unit is used to perform three-dimensional reconstruction of the marker points based on their coordinates and IDs to obtain the three-dimensional coordinates of the marker points. Based on the three-dimensional coordinates of at least three non-collinear marker points, the unit performs pose calculation on the target to obtain the second pose data. The first pose data and the second pose data are then fused to obtain the motion data of the target. This achieves the fusion of motion data obtained from perception data based on different modalities, improving the accuracy and robustness of the motion capture system in capturing the target.

[0096] In some embodiments, when the motion capture system includes a first image processing unit 106, the processing unit 102 is configured to perform at least one of the following:

[0097] Based on the coordinates of the marker points, perform three-dimensional reconstruction of the marker points;

[0098] Perform ID tracking on the marker points to obtain the marker point ID tracking results;

[0099] Keypoint detection is performed on visible light images to obtain keypoint information, which includes the coordinates and ID of the keypoints.

[0100] Motion attitude estimation of the target in the visible light image is performed to obtain the first pose data;

[0101] The marker points are reprojected onto the visible light image and compared with the IDs of the key points to verify the accuracy of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking.

[0102] For image frames that have lost marker information, the first pose data is used to fill in the missing information to obtain the motion data of the target being measured.

[0103] In some embodiments, the processing unit 102 is an embedded processing unit, such as a field programmable gate array (FPGA), embedded in the motion capture module 101, or it may exist in the motion capture system in the form of a host computer.

[0104] In some embodiments, the motion capture module 101 includes at least one non-visible light image sensor 103, which can sense the object under test from at least one viewpoint to obtain near-infrared or infrared band images. The first image processing unit 106 uses image processing technology to track the marker points in these near-infrared or infrared band images, calculates the coordinates of the marker points through spatial positioning, and outputs them to the processing unit 102. The processing unit 102 receives the coordinates of the marker points.

[0105] The processing unit 102 uses triangulation to perform multi-viewpoint three-dimensional reconstruction of the object under test based on the coordinates of the marker points, and obtains the three-dimensional coordinates of the marker points.

[0106] Processing unit 102 performs ID tracking on each marker to determine its positional changes in consecutive image frames or time series. By assigning a unique ID to each marker and tracking the ID of each marker, it enables the identification and association of the same marker during the dynamic capture of the target being measured. In some embodiments, processing unit 102 uses methods such as Kalman filtering to perform ID tracking on the markers, obtaining the marker ID tracking results, which include the three-dimensional coordinates of each marker and its corresponding image frame sequence or time series.

[0107] In some embodiments, the processing unit 102 uses a deep learning algorithm, such as YOLOv8, to perform key point detection on the visible light image and obtain the coordinates and ID of the key points, where the ID is a unique identifier of the key point.

[0108] In some embodiments, the processing unit 102 uses algorithms such as MediaPipe to estimate the motion posture of the target in the visible light image to obtain the first pose data.

[0109] In some embodiments, the processing unit 102 reprojects the marker points onto the visible light image based on the three-dimensional coordinates of the marker points, thereby converting the three-dimensional spatial coordinates of the marker points into two-dimensional reprojection coordinates. The projection of the marker points in the visible light image is compared with the key points detected in the visible light image, including comparing their coordinate positions and IDs, in order to verify the correctness of the marker point ID tracking or to identify the IDs of marker points that have lost tracking.

[0110] In some embodiments, verifying the correctness of the ID tracking results may include: verifying whether there are ambiguous ID matching for marker points, or verifying whether there are mismatched IDs for marker points. For example, each marker point may be reprojected onto a visible light image, and the projection distance features and the temporal motion features of the target being measured are compared with the key point information to verify the correctness of the ID tracking results for each marker point or to identify the IDs of marker points that have lost tracking. The projection distance feature involves projecting the spatial coordinates of the identified matching marker points onto the visible light image and performing distance matching with the key points detected in the visible light image. If the distance between the projected points of all marker points' 3D coordinates and their corresponding key points is the closest, then the matching is considered successful; if the projected points of marker points' 3D coordinates are close to multiple key points, or if it is difficult to distinguish the closest points, then the matching is considered ambiguous or incorrect.

[0111] For example, when the target being measured is a hand (such as a hand device, a human hand, etc.), the temporal motion characteristics of the target can include the temporal motion characteristics of each finger joint, as well as the constraint characteristics of the motion axis angle and motion direction of each finger joint.

[0112] For image frames lacking marker information, the missing marker information is filled in using the first pose data to obtain the motion data of the target. Here, the image frame refers to a non-visible light image. In some embodiments, the filling method can be direct filling or filtering, such as Kalman filtering.

[0113] In the above technical solution, the processing unit is used to perform three-dimensional reconstruction of the marker points based on their coordinates, and perform ID tracking on the marker points to obtain the marker point ID tracking results. It also performs keypoint detection on the visible light image to obtain the coordinates and IDs of the keypoints, estimates the motion posture of the target in the visible light image to obtain the first pose data, reprojects the marker points onto the visible light image, and compares them with the IDs of the keypoints to verify the correctness of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking. For image frames that have lost marker point information, the first pose data is used to fill in the missing information. This enables the acquisition of motion data of the target based on perception data from different modalities. Even in the case of lost marker point information, the motion data of the target can be obtained by filling in the missing information based on the first pose data obtained from motion posture estimation of the visible light image, thus improving the accuracy and robustness of the motion capture system in capturing the target.

[0114] Figure 5 is a fifth schematic diagram of the structure of a motion capture system provided in some embodiments of this application. As shown in Figure 5, in some embodiments, the motion capture system further includes: a second image processing unit 107, connected to a visible light image sensor 104, for performing one or more of the following processes on the visible light image: feature extraction, feature recognition, feature spatial positioning calculation, motion data calculation, and sending the processing results to the processing unit 102.

[0115] In some embodiments, feature extraction refers to extracting features from a visible light image that can be used for subsequent motion capture of the target under test. Features may include edges, corners, textures, etc. in the image. Feature recognition refers to the process of determining the specific identity or category of these features based on feature extraction. Feature spatial localization calculation refers to estimating the spatial position and pose of the target under test by recognizing and using features in the image. Motion data decomposition refers to the process of determining the motion data of the target under test.

[0116] The second image processing unit 107 processes the visible light image, and the processing unit 102 receives the processing result sent by the second image processing unit 107.

[0117] In some embodiments, the second image processing unit 107 is an embedded processing unit, such as a field programmable gate array (FPGA), embedded in the motion capture module 101, or it may exist in the motion capture system in the form of a host computer.

[0118] In the above technical solution, the second image processing unit can be used to perform feature extraction, feature recognition, feature space positioning calculation or motion data calculation on the visible image, and send the processing results to the processing unit. The second image processing unit can be an embedded processing unit or exist in the motion capture system in the form of a host computer, which improves the flexibility of the motion capture system.

[0119] In some embodiments, the second image processing unit 107 is specifically used for:

[0120] Keypoint detection is performed on visible light images to obtain keypoint information, which includes the coordinates and ID of the keypoints.

[0121] The motion attitude of the target in the visible light image is estimated to obtain the first pose data.

[0122] In some embodiments, the second image processing unit 107 employs deep learning algorithms, such as YOLOv8, to detect key points in the visible light image, obtaining the coordinates and IDs of the key points, where the ID is a unique identifier for the key point. In some embodiments, the second image processing unit 107 uses algorithms such as MediaPipe to estimate the motion pose of the target in the visible light image, obtaining first pose data. The pose data is an estimate of the motion data, and the motion data of the target includes the position data and pose data of the target in three-dimensional space.

[0123] In the above technical solution, the second image processing unit is used to detect key points in the visible light image to obtain the coordinates and ID of the key points, and to estimate the motion posture of the target in the visible light image to obtain the first pose data. This helps to obtain the motion data of the target based on the perception data of different modalities, thereby improving the accuracy and robustness of the motion capture system in capturing the target.

[0124] In some embodiments, where the motion capture system further includes a first image processing unit 106 and a second image processing unit 107, the processing unit 102 is configured to perform at least one of the following:

[0125] Based on the coordinates of the marker points, perform three-dimensional reconstruction of the marker points;

[0126] Perform ID tracking on the marker points to obtain the marker point ID tracking results;

[0127] The marker points are reprojected onto the visible light image and compared with the IDs of the key points to verify the accuracy of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking.

[0128] For image frames that have lost marker information, the first pose data is used to fill in the missing information to obtain the motion data of the target being measured.

[0129] In some embodiments, the motion capture module 101 includes at least one non-visible light image sensor 103, which can sense the object under test from at least one viewpoint to obtain near-infrared or infrared images. A first image processing unit 106 uses image processing technology to track marker points in these near-infrared or infrared images, calculates the coordinates of the marker points through spatial positioning, and outputs them to the processing unit 102, i.e., the processing unit 102 receives the marker point coordinates. Based on the marker point coordinates, the processing unit 102 uses triangulation to perform multi-viewpoint 3D reconstruction of the object under test to obtain the 3D coordinates of the marker points.

[0130] In some embodiments, the processing unit 102 tracks each marker point to determine its positional changes in consecutive image frames or time series. By assigning a unique ID to each marker point and tracking the ID of each marker point, the same marker point can be identified and associated during the dynamic capture of the target being measured. In some embodiments, the processing unit 102 uses methods such as Kalman filtering to track the IDs of the marker points, obtaining the marker point ID tracking results, which include the three-dimensional coordinates of each marker point and its corresponding image frame sequence or time series.

[0131] In some embodiments, the processing unit 102 reprojects the marker points onto the visible light image based on the three-dimensional coordinates of the marker points, thereby converting the three-dimensional spatial coordinates of the marker points into two-dimensional reprojection coordinates. The projection of the marker points in the visible light image is compared with the key points detected in the visible light image, including comparing their coordinate positions and IDs, in order to verify the correctness of the marker point ID tracking or to identify the IDs of marker points that have lost tracking.

[0132] In some embodiments, verifying the correctness of the ID tracking results may include: verifying whether there are ambiguous ID matching for marker points, or verifying whether there are mismatched IDs for marker points. For example, each marker point may be reprojected onto a visible light image, and the projection distance features and the temporal motion features of the target being measured are compared with the key point information to verify the correctness of the ID tracking results for each marker point or to identify the IDs of marker points that have lost tracking. The projection distance feature involves projecting the spatial coordinates of the identified matching marker points onto the visible light image and performing distance matching with the key points detected in the visible light image. If the distance between the projected points of all marker points' 3D coordinates and their corresponding key points is the closest, then the matching is considered successful; if the projected points of marker points' 3D coordinates are close to multiple key points, or if it is difficult to distinguish the closest points, then the matching is considered ambiguous or incorrect.

[0133] For image frames lacking marker information, the missing marker information is filled in using the first pose data to obtain the motion data of the target. Here, the image frame refers to a non-visible light image. In some embodiments, the filling method can be direct filling or filtering, such as Kalman filtering.

[0134] In the above technical solution, the processing unit is used to perform three-dimensional reconstruction of the marker points based on their coordinates, perform ID tracking on the marker points to obtain the marker point ID tracking results, reproject the marker points onto the visible light image, and compare them with the IDs of key points to verify the correctness of the marker point ID tracking results or to perform ID recognition on marker points that have lost tracking. For image frames that have lost marker point information, pose data is used to fill in the missing information. This enables the acquisition of motion data of the target object based on perception data of different modalities. In the case of lost marker point information, the motion data of the target object can also be obtained by filling in the missing information based on the first pose data obtained by motion attitude estimation of the visible light image. This improves the accuracy and robustness of the motion capture system in capturing the motion of the target object.

[0135] Figure 6 is a schematic diagram of the structure of a motion capture system provided in some embodiments of this application. As shown in Figure 6, in some embodiments, the motion capture module includes:

[0136] At least one inertial sensor 108 is disposed on the target being measured and is used to measure the inertial data during the motion of the target being measured and output the inertial data to the processing unit.

[0137] In some embodiments, a non-visible light image sensor 103 is used to sense the target under test in the near-infrared or infrared band and output images in the near-infrared or infrared band. Since the light in the near-infrared and infrared bands is invisible to the human eye, the non-visible light image sensor is not limited by visible light conditions and is more suitable for environments with less than ideal visible light conditions. The motion capture module includes at least one non-visible light image sensor 103, which improves the applicability of the motion capture system under different lighting conditions. A visible light image sensor 104 is used to sense the target under test in the visible light band and output visible light images. Under good lighting conditions, the visible light image sensor 104 can output high-resolution color images of the target under test. An inertial sensor 108 is disposed on the target under test and measures the inertial data during the motion of the target under test, including data reflecting changes in motion state and direction, such as triaxial angular velocity, triaxial acceleration, triaxial magnetometer data, etc. By fusing inertial data, the robustness against occlusion in the motion calculation of the target under test can be improved.

[0138] In some embodiments, the inertial sensor installed on the target under test is bound to the target under test through device identification information such as serial number (SN).

[0139] In some embodiments, the motion capture module 101 includes at least one non-visible light image sensor 103, at least one visible light image sensor 104, and at least one inertial sensor 108. This allows the motion capture module 101 to perceive the target in optical, visual, and inertial modes. The optical mode corresponds to the near-infrared or infrared image output by the non-visible light image sensor 103, the visual mode corresponds to the visible light image output by the visible light image sensor 104, and the inertial mode corresponds to the inertial data output by the inertial sensor. This improves the comprehensiveness and accuracy of perceiving the target. Motion data is obtained based on the perception data from different modes, which also improves the accuracy of the motion capture system in capturing the motion of the target.

[0140] The processing unit 102 is used to process sensing data from different modalities, such as data matching, fusion, and analysis, to obtain motion data of the target being measured. The motion data describes the target's motion state in three-dimensional space, including the object's motion data, thereby achieving motion capture of the target. In some embodiments, motion data is obtained based on sensing data from different modalities, realizing the fusion of sensing data from different modalities and improving the accuracy of the motion capture system in capturing the motion of the target. A single-modal sensor may be affected by occlusion or other factors, resulting in inaccurate output sensing data. Fusion of sensing data from multiple modalities can improve the robustness of the motion capture system.

[0141] In the above technical solution, the motion capture system includes a motion capture module and a processing unit. The motion capture module includes sensors of two or more modalities for sensing the target and obtaining sensing data of different modalities. The processing unit is connected to the motion capture module and processes the sensing data of different modalities to obtain the motion data of the target, thus realizing the motion capture of the target. The motion capture module includes at least one inertial sensor, which is set on the target to measure the inertial data during the movement of the target. Based on the sensing data of different modalities, the motion data of the target is obtained, which improves the accuracy and robustness of the motion capture system in capturing the target.

[0142] Figure 7 is a schematic diagram of the structure of a motion capture system provided in some embodiments of this application. As shown in Figure 7, in some embodiments, the motion capture module 101 further includes:

[0143] At least one non-visible light image sensor 103 is used to sense the target in the near-infrared or infrared band and output an image in the near-infrared or infrared band; and / or,

[0144] At least one visible light image sensor 104 is provided, which is used to sense the target under test in the visible light band and output a visible light image.

[0145] In some embodiments, the motion capture module 101 includes at least one inertial sensor 103 and at least one non-visible light image sensor 103;

[0146] Alternatively, it may include at least one inertial sensor 103 and at least one visible light image sensor 104;

[0147] Alternatively, it may include at least one inertial sensor 103, at least one non-visible light image sensor 103, and at least one visible light image sensor 104.

[0148] In some embodiments, a non-visible light image sensor 103 is used to sense the target under test in the near-infrared or infrared band and output images in the near-infrared or infrared band. Since the light in the near-infrared and infrared bands is invisible to the human eye, the non-visible light image sensor is not limited by visible light conditions and is more suitable for environments with less than ideal visible light conditions. The motion capture module includes at least one non-visible light image sensor 103, which improves the applicability of the motion capture system under different lighting conditions. A visible light image sensor 104 is used to sense the target under test in the visible light band and output visible light images. Under good lighting conditions, the visible light image sensor 104 can output high-resolution color images of the target under test. An inertial sensor 103 is disposed on the target under test and measures the inertial data during the motion of the target under test, including data reflecting changes in motion state and direction, such as triaxial angular velocity, triaxial acceleration, triaxial magnetometer data, etc. By fusing inertial data, the robustness against occlusion in the motion calculation of the target under test can be improved.

[0149] In some embodiments, the inertial sensor installed on the target under test is bound to the target under test through device identification information such as serial number (SN).

[0150] In some embodiments, the motion capture module 101 includes at least one non-visible light image sensor 103, at least one visible light image sensor 104, and at least one inertial sensor 103. This allows the motion capture module 101 to perceive the target from optical, visual, and inertial modes. The optical mode corresponds to the near-infrared or infrared image output by the non-visible light image sensor 103, the visual mode corresponds to the visible light image output by the visible light image sensor 104, and the inertial mode corresponds to the inertial data output by the inertial sensor. This improves the comprehensiveness and accuracy of perceiving the target. Motion data is obtained based on the perception data from different modes, which also improves the accuracy of the motion capture system in capturing the motion of the target.

[0151] The processing unit 102 is used to process sensing data from different modalities, such as data matching, fusion, and analysis, to obtain motion data of the target being measured. The motion data describes the target's motion state in three-dimensional space, including its pose data, thereby achieving motion capture of the target. In some embodiments, motion data is obtained based on sensing data from different modalities, realizing the fusion of sensing data from different modalities and improving the accuracy of the motion capture system in capturing the motion of the target. A single-modal sensor may be affected by occlusion or other factors, resulting in inaccurate output sensing data. Fusion of sensing data from multiple modalities can improve the robustness of the motion capture system.

[0152] In the above technical solution, the motion capture system includes a motion capture module and a processing unit. The motion capture module includes sensors of two or more modalities for sensing the target and obtaining sensing data of different modalities. The processing unit is connected to the motion capture module and processes the sensing data of different modalities to obtain the motion data of the target, thus realizing the motion capture of the target. The motion capture module includes at least one inertial sensor, which is set on the target to measure the inertial data during the movement of the target. Based on the sensing data of different modalities, the motion data of the target is obtained, which improves the accuracy and robustness of the motion capture system in capturing the target.

[0153] Figure 8 is a schematic diagram of the structure of a motion capture system provided in some embodiments of this application. As shown in Figure 8, in some embodiments, when the motion capture system includes at least one non-visible light image sensor 103, the motion capture system also includes at least one near-infrared or infrared band supplementary light source 105 for illuminating the target being measured;

[0154] At least one marker is arranged on the target being measured. Under the illumination of the supplementary light source 105, the marker has a contrast with the background in the imaging of the non-visible light image sensor 103.

[0155] The near-infrared or infrared band supplementary light source 105 is used to provide near-infrared or infrared band light to illuminate the target under test, so that the non-visible light image sensor 103 can perceive the target under test more clearly, improve the quality of the near-infrared or infrared band image output by the non-visible light image sensor 103, and also enable the non-visible light image sensor 103 to still perceive the target under near-infrared or infrared band lighting conditions that are not ideal.

[0156] At least one marker is placed on the target being measured. The marker is a specially designed mark or luminous point, also known as a marker. Under the illumination of the supplementary light source 105, the marker will create a contrast with the background in the imaging of the non-visible light image sensor, making the marker stand out more in the near-infrared or infrared band image output by the non-visible light image sensor 103. This contrast with the surrounding background makes the marker easier to identify and track in subsequent image processing and analysis, improving the accuracy of marker identification and positioning.

[0157] In the above technical solution, when the motion capture system includes at least one non-visible light image sensor, the motion capture system includes at least one near-infrared or infrared band supplementary light source for illuminating the target under test, and at least one marker point is arranged on the target under test. Under the illumination of the supplementary light source, the marker point has a contrast with the background in the imaging of the non-visible light image sensor, making the marker point more prominent, improving the accuracy of identifying and locating the marker point in the imaging of the non-visible light image sensor, and thus improving the accuracy of the motion capture system in motion capture of the target under test.

[0158] Figure 9 is a schematic diagram of the structure of a motion capture system provided in some embodiments of this application. As shown in Figure 9, in some embodiments, the motion capture system further includes: a first image processing unit 106, connected to a non-visible light image sensor 103, used to track and spatially locate markers in near-infrared or infrared images, and output the coordinates of the markers to the processing unit 102.

[0159] In some embodiments, the motion capture module 101 in the motion capture system includes at least one non-visible light image sensor 103, at least one visible light image sensor 104, and at least one inertial sensor 103, for sensing the target under test and obtaining near-infrared or infrared images, visible light images, and inertial data. In some embodiments, the processing unit 102 processes the near-infrared or infrared images, visible light images, and inertial data to obtain motion data of the target under test.

[0160] In some embodiments, the motion capture system further includes a first image processing unit 106, which uses image processing technology to track marker points in near-infrared or infrared band images and calculates the coordinates of the marker points through spatial positioning. The processing unit 102 receives the coordinates of the marker points output by the first image processing unit 106, thereby improving the flexibility of the motion capture system.

[0161] In some embodiments, the first image processing unit 106 is an embedded processing unit, such as a field programmable gate array (FPGA), embedded in the motion capture module 101 and connected to the non-visible light image sensor 103.

[0162] In the above technical solution, the motion capture system also includes a first image processing unit, which is connected to a non-visible light image sensor. The first image processing unit is used to track and spatially locate markers in near-infrared or infrared images and output the coordinates of the markers to the processing unit, thereby improving the flexibility of the motion capture system.

[0163] Figure 10 is a schematic diagram of the structure of a motion capture system provided in some embodiments of this application. As shown in Figure 10, in some embodiments, where the motion capture system includes at least one visible light image sensor, the motion capture system further includes: a second image processing unit 107, connected to the visible light image sensor 104, for:

[0164] Keypoint detection is performed on visible light images to obtain keypoint information, which includes the coordinates and ID of the keypoints.

[0165] Motion attitude estimation is performed on the target in the visible light image to obtain the first pose data of the target.

[0166] The second image processing unit 107 is connected to the visible light image sensor 104 and is used to perform one or more of the following processes on the visible light image: feature extraction, feature recognition, feature spatial positioning calculation, motion data calculation, and send the processing results to the processing unit 102.

[0167] Feature extraction refers to extracting features from visible light images that can be used for subsequent motion capture of the target under test. These features may include edges, corners, textures, etc. Feature recognition refers to the process of determining the specific identity or category of these features based on feature extraction. Feature spatial localization calculation refers to estimating the spatial position and pose of the target under test by recognizing and using features in the image. Motion data solution refers to the process of determining the motion pose data of the target under test, which includes the position and pose data of the target under test in three-dimensional space.

[0168] The second image processing unit 107 processes the visible light image, and the processing unit 102 receives the processing result sent by the second image processing unit 107.

[0169] In some embodiments, the second image processing unit 107 is an embedded processing unit, such as a field programmable gate array (FPGA), embedded in the motion capture module 101, or it may exist in the motion capture system in the form of a host computer.

[0170] In the above technical solution, the second image processing unit is used to perform feature extraction, feature recognition, feature spatial positioning calculation or motion data calculation on the visible light image, and send the processing results to the processing unit. The second image processing unit can be an embedded processing unit or exist in the motion capture system in the form of a host computer, which improves the flexibility of the motion capture system.

[0171] In some embodiments, the second image processing unit 107 employs deep learning algorithms, such as YOLOv8, to perform keypoint detection on the visible light image, obtaining the coordinates and IDs of the keypoints, where the ID is a unique identifier for the keypoint. In some embodiments, the second image processing unit 107 may use algorithms such as MediaPipe to estimate the motion pose of the target in the visible light image, obtaining first pose data.

[0172] The second image processing unit is used to detect key points in the visible light image, obtain the coordinates and IDs of the key points, estimate the motion pose of the target in the visible light image, and obtain the first pose data. The first pose data can be used to fuse with pose data obtained from perception data based on other modalities to obtain the motion data of the target, thereby improving the accuracy and robustness of the motion capture system in capturing the target.

[0173] As shown in Figure 10, in the motion capture module 101 of the motion capture system, which includes at least one non-visible light image sensor 103, at least one visible light image sensor 104, and at least one inertial sensor 103, and the motion capture system further includes a first image processing unit 106 and a second image processing unit 107, the processing unit 102 is used to perform at least one of the following:

[0174] Based on the coordinates of the marker points, perform three-dimensional reconstruction of the marker points;

[0175] Perform ID tracking on the marker points to obtain the marker point ID tracking results;

[0176] Calculate the second pose data of the target based on the coordinates of at least three non-collinear marker points on the target.

[0177] Based on the inertial data, the motion attitude of the target under test is estimated to obtain the third pose data of the target under test.

[0178] The marker points are reprojected onto the visible light image and compared with the IDs of the key points to verify the accuracy of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking.

[0179] The motion data of the target under test are obtained by fusing the first pose data, the second pose data and the third pose data.

[0180] In some embodiments, the motion capture module 101 includes at least one non-visible light image sensor 103, which can sense the object under test from at least one viewpoint to obtain near-infrared or infrared images. The first image processing unit 106 tracks the marker points in these near-infrared or infrared images through image processing technology, calculates the coordinates of the marker points through spatial positioning, and outputs them to the processing unit 102, that is, the processing unit 102 receives the coordinates of the marker points.

[0181] The processing unit 102 uses triangulation to perform multi-viewpoint three-dimensional reconstruction of the object under test based on the coordinates of the marker points, and obtains the three-dimensional coordinates of the marker points.

[0182] Processing unit 102 tracks each marker point to determine its positional changes in consecutive image frames or time series. By assigning a unique ID to each marker point and tracking each marker point's ID, it enables the identification and association of the same marker point during the dynamic capture of the target being measured. In some embodiments, processing unit 102 uses methods such as Kalman filtering to track the marker point IDs, obtaining the marker point ID tracking results, which include the three-dimensional coordinates of each marker point and its corresponding image frame sequence or time series.

[0183] In some embodiments, the processing unit 102 calculates the pose of the target under test based on the three-dimensional coordinates of at least three non-collinear marker points to obtain second pose data PO. For example, an object coordinate system can be constructed using the three marker points, with one marker point as the origin, the line connecting the origin and another marker point as the x-axis, and the plane defined by the three marker points as the xy-plane. The z-axis direction is then determined through vector cross product operations, thereby deriving the rotation matrix and translation vector to obtain the position and pose data of the target under test. Alternatively, the second pose data of the target under test can be calculated using the known three-dimensional coordinates of three or more non-collinear marker points and their two-dimensional projection coordinates in the image. In some embodiments, features of the target under test, such as joint topology or continuous motion constraints, can be combined to optimize the pose calculation process and improve the accuracy and reliability of the pose calculation.

[0184] In some embodiments, the processing unit 102 uses a deep learning algorithm, such as YOLOv8, to perform key point detection on the visible light image and obtain the coordinates and ID of the key points, where the ID is a unique identifier of the key point.

[0185] In some embodiments, the processing unit 102 uses algorithms such as MediaPipe to estimate the motion pose of the target in the visible light image to obtain the first pose data PA.

[0186] In some embodiments, the processing unit 102 uses a complementary filtering algorithm based on inertial data to estimate the motion attitude of the target under test and obtains the third pose data of the target under test. In this third pose data, the spatial positioning is replaced by the displacement vector T obtained from the second pose data PO.

[0187] In some embodiments, the processing unit 102 reprojects the marker points onto a visible light image based on their three-dimensional coordinates, converting the three-dimensional spatial coordinates of the marker points into two-dimensional reprojected coordinates. The projection of the marker points in the visible light image is then compared with the key points detected in the visible light image, including comparing their coordinate positions and IDs to verify the accuracy of the marker point ID tracking results. This verifies the correctness of the marker ID tracking or performs ID recognition on marker points that have lost tracking. Verifying the correctness of the ID tracking results includes checking for ambiguous or mismatched marker point IDs.

[0188] The first pose data is determined based on a visible light image, the second pose data is determined based on a near-infrared or infrared image, and the third pose data is determined based on inertial data. In some embodiments, the processing unit 102 uses methods such as Kalman filtering to fuse the second pose data PO, the first pose data PA, and the third pose data PB to obtain the motion data P of the target. The motion data of the target describes the motion posture of the target in three-dimensional space, thereby realizing the motion capture of the target. This method can solve the problem of marker ID tracking errors or loss of marker information, and can still reliably estimate the motion posture using visible light images and inertial sensors even after marker loss, avoiding complete loss of motion data.

[0189] In the above technical solution, the processing unit performs 3D reconstruction of the marker points based on their coordinates, performs ID tracking on the marker points to obtain the marker point ID tracking results, calculates the second pose data of the target based on the coordinates of at least three non-collinear marker points on the target, estimates the motion attitude of the target based on inertial data to obtain the third pose data of the target, reprojects the marker points onto the visible light image, and compares them with the IDs of key points to verify the correctness of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking. This achieves the acquisition of pose data of the target based on perception data from different modalities, and the fusion of pose data obtained from perception data from various modalities to obtain motion data of the target, thereby improving the accuracy and robustness of the motion capture system in capturing motion of the target.

[0190] In some embodiments, the motion capture module 101 further includes optical lenses disposed in front of each sensor. The sensors are used to perceive the target under test through the optical lenses to obtain perception data of different modalities. The optical lenses can be used to adjust and focus the light captured by the sensors so that each sensor can perceive the target under test more clearly and accurately. The sensors and optical lenses work in coordination to obtain perception data of different modalities of the target under test.

[0191] The motion capture system provided in the above embodiments of this application realizes motion capture through optical-visual-inertial multimodal fusion, and can be used for high-precision, high-robust motion tracking, positioning and kinematic calculation of multiple targets.

[0192] The motion capture method provided in this application embodiment can be executed by a motion capture system or a functional module or entity within the motion capture system that can implement the motion capture method. The motion capture method provided in this application embodiment is described below using a motion capture system as the execution subject as an example.

[0193] Figure 11 is a schematic flowchart of a motion capture method provided in some embodiments of this application. As shown in Figure 11, the motion capture method, based on the above-mentioned motion capture system, includes steps 1110 and 1120.

[0194] Step 1110: Perceive the target under test and obtain perception data of different modalities.

[0195] In some embodiments, the target is sensed using a motion capture module, which includes sensors of various modalities. These sensors can be of different types, such as non-visible light image sensors and visible light image sensors. Each modal sensor can sense the target and obtain specific modal sensing data; for example, a near-infrared image sensor senses the target and obtains a near-infrared image, while a visible light image sensor senses the target and obtains a visible light image. By using sensors of multiple modalities, the target can be sensed from multiple modalities, improving the comprehensiveness and accuracy of target sensing.

[0196] Step 1120: Process the sensing data of different modalities to obtain the motion data of the target being measured.

[0197] In some embodiments, sensing data from different modalities are processed, such as through data matching, fusion, and analysis, to ultimately obtain the motion data of the target being measured. This motion data includes the target's position and orientation data in three-dimensional space, thereby achieving motion capture of the target. In some embodiments, motion data is obtained based on sensing data from different modalities, realizing the fusion of sensing data from different modalities. This improves the accuracy of the motion capture system in capturing the target. A single-modal sensor may be affected by occlusion or other factors, resulting in inaccurate output sensing data. Fusion of sensing data from multiple modalities can improve the robustness of the motion capture system.

[0198] In the above technical solution, the target under test is perceived to obtain perception data of different modalities, and the perception data of different modalities is processed to obtain motion data of the target under test. This realizes the fusion of perception data of different modalities and improves the accuracy and robustness of motion capture method for motion capture of target under test.

[0199] In some embodiments, the target under test is sensed to obtain sensing data in different modalities, including:

[0200] The target under test is sensed in the near-infrared or infrared band, and an image in the near-infrared or infrared band is output, wherein the image in the near-infrared or infrared band includes at least one marker point.

[0201] The coordinates of the marker points are obtained by tracking and spatially locating the marker points in near-infrared or infrared images.

[0202] It senses the target in the visible light band and outputs a visible light image.

[0203] Keypoint detection is performed on visible light images to obtain keypoint information, which includes the coordinates and ID of the keypoints.

[0204] Motion attitude estimation of the target in the visible light image is performed to obtain the first pose data;

[0205] Match the coordinates of key points with the coordinates of marker points to determine the ID of the marker points.

[0206] In some embodiments, the target under test is sensed in the near-infrared or infrared band and an image in the near-infrared or infrared band is output. Since the light in the near-infrared and infrared bands is invisible to the human eye, the sensing of the target under test in the near-infrared or infrared bands is not limited by visible light conditions and can be performed even in environments where visible light conditions are not ideal, thus improving the applicability of the motion capture method under different lighting conditions.

[0207] In some embodiments, the target under test is sensed in the visible light band and a visible light image is output. Under good lighting conditions, sensing the target under test in the visible light band can output a high-resolution color visible light image.

[0208] In some embodiments, at least one marker is arranged on the target under test. These markers contrast with the background in near-infrared or infrared images, appearing as bright spots. The markers in the near-infrared or infrared images are tracked using image processing techniques, and their coordinates are calculated through spatial positioning.

[0209] In some embodiments, deep learning algorithms, such as YOLOv8, are used to detect key points in visible light images to obtain the coordinates and IDs of the key points, where the ID is a unique identifier for the key point. In other embodiments, algorithms such as MediaPipe can be used to estimate the motion pose of the target in the visible light image to obtain first pose data.

[0210] The coordinates of marker points are matched with the coordinates of key points to establish a correspondence between them, and a unique ID is assigned to each marker point. In some embodiments, assigning an ID to a marker point includes: comparing the coordinates of key points and marker points, determining the correspondence between marker points and key points by finding the nearest match, and assigning a consistent ID value to marker points that match key points in the same spatial location, so that each marker point has a unique and temporally consistent ID.

[0211] In the above technical solution, the target is sensed in the near-infrared or infrared band, and an image in the near-infrared or infrared band is output. The image in the near-infrared or infrared band includes at least one marker point. The marker point in the near-infrared or infrared band image is tracked and its spatial positioning is calculated to obtain the marker point coordinates. The target is sensed in the visible light band, and a visible light image is output. Key point detection is performed on the visible light image to obtain the coordinates and ID of the key point. The motion attitude of the target in the visible light image is estimated to obtain the first pose data. The coordinates of the key point and the coordinates of the marker point are matched to determine the ID of the marker point. This helps to obtain the motion data of the target based on the sensing data of different modalities, thereby improving the accuracy and robustness of the motion capture system for motion capture of the target.

[0212] In some embodiments, sensing data from different modalities are processed to obtain motion data of the target being measured, including:

[0213] Based on the coordinates and ID of the marker, the marker is reconstructed in three dimensions to obtain its three-dimensional coordinates;

[0214] Based on the three-dimensional coordinates of at least three non-collinear marker points, the pose of the target under test is calculated to obtain the second pose data;

[0215] The motion data of the target object are obtained by fusing the first pose data and the second pose data.

[0216] In some embodiments, the markers are reconstructed in three dimensions using triangulation based on their coordinates and IDs to obtain their three-dimensional coordinates; the pose of the target is calculated based on the three-dimensional coordinates of at least three non-collinear markers to obtain second pose data.

[0217] The second pose data of the target is obtained by calculating the pose of the target based on the 3D coordinates of at least three non-collinear marker points. For example, an object coordinate system can be constructed using the three marker points, with one marker point as the origin, the line connecting the origin and another marker point as the x-axis, and the plane defined by the three marker points as the xy-plane. The z-axis direction is then determined through vector cross product, and the rotation matrix and translation vector are derived to obtain the position and pose data of the target. Alternatively, the second pose data of the target can be calculated using the known 3D coordinates of three or more non-collinear marker points and their 2D projection coordinates in the image. In some embodiments, features of the target, such as joint topology or continuous motion constraints, can be combined to optimize the pose calculation process and improve the accuracy and reliability of the pose calculation.

[0218] In some embodiments, fusion algorithms such as Kalman filtering are used to fuse the first pose data and the second pose data to obtain the motion data of the target being measured. This realizes the fusion of motion data obtained from perception data based on different modalities. The fusion calculation can effectively improve the anti-occlusion of the target motion solution while ensuring that the accuracy loss is controllable, thereby improving the accuracy and robustness of the motion capture system in capturing the target being measured.

[0219] In the above technical solution, the marker points are reconstructed in three dimensions based on their coordinates and IDs to obtain their three-dimensional coordinates. Based on the three-dimensional coordinates of at least three non-collinear marker points, the pose of the target under test is calculated to obtain the second pose data. The first pose data and the second pose data are then fused to obtain the motion data of the target under test. This achieves the fusion of motion data obtained from perception data based on different modalities, improving the accuracy and robustness of the motion capture method for motion capture of the target under test.

[0220] In some embodiments, the target under test is sensed to obtain sensing data in different modalities, including:

[0221] The target under test is sensed in the near-infrared or infrared band, and an image in the near-infrared or infrared band is output, wherein the image in the near-infrared or infrared band includes at least one marker point.

[0222] The coordinates of the marker points are obtained by tracking and spatially locating the marker points in near-infrared or infrared images.

[0223] The target under test is sensed in the visible light band, and a visible light image is output.

[0224] In some embodiments, the sensing data of different modalities are processed to obtain the motion data of the target under test, including: determining the motion data of the target under test based on the coordinates of the marker points and the visible light image;

[0225] Based on the coordinates of the marker points and the visible light image, the motion data of the target under test is determined, including:

[0226] Based on the coordinates of the marker points, perform three-dimensional reconstruction of the marker points;

[0227] Perform ID tracking on the marker points to obtain the marker point ID tracking results;

[0228] Keypoint detection is performed on visible light images to obtain keypoint information, which includes the coordinates and ID of the keypoints.

[0229] Motion attitude estimation of the target in the visible light image is performed to obtain the first pose data;

[0230] The marker points are reprojected onto the visible light image and compared with the IDs of the key points to verify the accuracy of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking.

[0231] For image frames that have lost marker information, the first pose data is used to fill in the missing information to obtain the motion data of the target being measured.

[0232] Based on the coordinates of the marker points, 3D reconstruction of the marker points is performed, and ID tracking is conducted to obtain the marker point ID tracking results. Keypoint detection is performed on the visible light image to obtain the coordinates and IDs of the keypoints. Motion pose estimation of the target in the visible light image is performed to obtain the first pose data. The marker points are reprojected onto the visible light image and compared with the IDs of the keypoints to verify the correctness of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking. For image frames with missing marker point information, the first pose data is used to fill in the missing information. This enables the acquisition of motion pose data of the target based on perception data from different modalities. Even when marker point information is lost, the pose estimation results obtained from motion pose estimation of the visible light image can be used to fill in the missing information, thereby obtaining the motion data of the target and improving the accuracy and robustness of the motion capture system in capturing the motion of the target.

[0233] In the above technical solution, the target under test is sensed in the near-infrared or infrared band, and an image in the near-infrared or infrared band is output. The image in the near-infrared or infrared band includes at least one marker point. The marker point in the near-infrared or infrared band image is tracked and spatially located to obtain the marker point coordinates. The target under test is sensed in the visible light band, and a visible light image is output. Based on the marker point coordinates and the visible light image, the motion data of the target under test is obtained. This realizes the fusion of sensing data from different modalities and improves the accuracy and robustness of the motion capture method.

[0234] In some embodiments, the target under test is sensed to obtain sensing data in different modalities, including:

[0235] Measure the inertial data of the target object during its motion.

[0236] In some embodiments, inertial data during the motion of the target is measured by inertial sensors installed on the target, including data reflecting changes in motion state and direction, such as triaxial angular velocity, triaxial acceleration, and triaxial magnetometer data. By fusing inertial data, the robustness against occlusion in the motion calculation of the target can be improved.

[0237] In some embodiments, sensing the target to obtain sensing data in different modalities further includes:

[0238] The target is sensed in the near-infrared or infrared band, and an image in the near-infrared or infrared band is output, wherein the image in the near-infrared or infrared band includes at least one marker point; and / or,

[0239] The target under test is sensed in the visible light band, and a visible light image is output.

[0240] In some embodiments, sensing data from different modalities are processed to obtain motion data of the target being measured, including:

[0241] The coordinates of the marker points are obtained by tracking and spatially locating the marker points in near-infrared or infrared images.

[0242] Based on the coordinates of the marker points, perform three-dimensional reconstruction of the marker points;

[0243] Perform ID tracking on the marker points to obtain the marker point ID tracking results;

[0244] Calculate the first pose data of the target based on the coordinates of at least three non-collinear marker points on the target.

[0245] Keypoint detection is performed on visible light images to obtain keypoint information, which includes the coordinates and ID of the keypoints.

[0246] Motion pose estimation is performed on the target in the visible light image to obtain the second pose data of the target;

[0247] Based on the inertial data, the motion attitude of the target under test is estimated to obtain the third pose data of the target under test.

[0248] The marker points are reprojected onto the visible light image and compared with the IDs of the key points to verify the accuracy of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking.

[0249] The motion data of the target under test are obtained by fusing the first pose data, the second pose data and the third pose data.

[0250] The system performs 3D reconstruction of marker points and ID tracking to obtain the marker point ID tracking results. Based on the coordinates of at least three non-collinear marker points on the target, it calculates the first pose data of the target. Keypoint detection is performed on the visible light image to obtain the coordinates and IDs of the keypoints. Motion pose estimation is performed on the target in the visible light image to obtain the second pose data. Based on inertial data, motion pose estimation is performed on the target to obtain the second pose data of the target. The marker points are reprojected onto the visible light image and compared with the IDs of the keypoints to verify the accuracy of the marker point ID tracking results or to identify the IDs of lost marker points. This system achieves pose estimation results for the target based on perception data from different modalities, and fuses the pose estimation results obtained from perception data from each modality to obtain the motion data of the target. This improves the accuracy and robustness of the motion capture system for target motion capture.

[0251] In the above technical solution, the target under test is sensed to obtain sensing data of different modalities, including sensing the target under test in the near-infrared or infrared band and outputting near-infrared or infrared band images, wherein the near-infrared or infrared band images include at least one marker point, tracking and spatial positioning calculation of the marker point in the near-infrared or infrared band images to obtain the marker point coordinates, sensing the target under test in the visible light band and outputting visible light images, and measuring the inertial data of the target under test during its motion process. The sensing data of different modalities are processed to obtain the motion data of the target under test, realizing the fusion of sensing data of different modalities and improving the accuracy and robustness of the motion capture method.

[0252] For a better understanding of the motion capture method provided in the embodiments of this application, please refer to the foregoing description of the motion capture system, which achieves the same technical effect. To avoid repetition, it will not be described again here.

[0253] Figure 12 is a schematic diagram of the structure of an electronic device provided in some embodiments of this application. In some embodiments, as shown in Figure 12, this application also provides an electronic device 1200, including a processor 1201, a memory 1202, and a computer program stored in the memory 1202 and executable on the processor 1201. When the program is executed by the processor 1201, it implements the various processes of the above-described motion capture method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0254] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0255] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described motion capture method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0256] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0257] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described motion capture method.

[0258] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0259] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described motion capture method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0260] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0261] This application constructs a motion capture system, which includes motion capture modules. Each motion capture module includes sensors with multiple modalities. The sensors are used to sense the target being measured and obtain sensing data of different modalities. A processing unit is connected to the motion capture module to process the sensing data of different modalities to obtain the motion data of the target being measured. This realizes the motion capture of the target being measured. By obtaining motion data based on sensing data of different modalities, the accuracy and robustness of the motion capture system in capturing the target being measured are improved. It has promising prospects for engineering implementation.

[0262] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0263] From the above description of the embodiments, it can be understood that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0264] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Under the guidance of this application, many other forms can be made without departing from the spirit and scope of the claims, and all of them are within the protection scope of this application.

[0265] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0266] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A motion capture system, comprising: The motion capture module includes sensors of various modalities, used to sense the target under test and obtain sensing data of different modalities; The processing unit, connected to the motion capture module, is used to process the sensing data of the different modalities to obtain the motion data of the target being measured.

2. The motion capture system according to claim 1, wherein, The motion capture module includes: At least one non-visible light image sensor, the non-visible light image sensor being used to sense the target under test in the near-infrared or infrared band and output an image in the near-infrared or infrared band; At least one visible light image sensor, the visible light image sensor being used to sense the target under test in the visible light band and output a visible light image.

3. The motion capture system according to claim 2, wherein, The motion capture system also includes at least one near-infrared or infrared light source for illuminating the target being measured. At least one marker is arranged on the target being measured. Under the illumination of the supplementary light source, the marker has a contrast with the background in the imaging of the non-visible light image sensor.

4. The motion capture system according to claim 3, wherein, The motion capture system also includes: The first image processing unit is connected to the non-visible light image sensor and is used to track and spatially locate the marker points in the near-infrared or infrared band image, and output the marker point coordinates to the processing unit.

5. The motion capture system according to claim 4, wherein, The first image processing unit is also connected to the visible light image sensor; the first image processing unit is further configured to: The visible light image is subjected to one or more of the following processes: feature extraction, feature recognition, feature spatial positioning calculation, motion data solution, and the processing results are sent to the processing unit.

6. The motion capture system according to claim 5, wherein, The first image processing unit is specifically used for: Key point detection is performed on the visible light image to obtain key point information, which includes the coordinates and ID of the key points; The motion attitude of the target in the visible light image is estimated to obtain the first pose data; The coordinates of the key points and the coordinates of the marker points are matched to determine the ID of the marker points; The coordinates of the marker point, the ID of the marker point, and the first pose data are sent to the processing unit.

7. The motion capture system according to claim 6, wherein, The processing unit is used for: Based on the coordinates and ID of the marker point, a 3D reconstruction is performed on the marker point to obtain its 3D coordinates; Based on the three-dimensional coordinates of at least three non-collinear marker points, the pose of the target under test is calculated to obtain the second pose data; The motion data of the target under test are obtained by fusing the first pose data and the second pose data.

8. The motion capture system according to claim 4, wherein, The processing unit is configured to perform at least one of the following: Based on the coordinates of the marker point, a three-dimensional reconstruction of the marker point is performed; The ID tracking of the marker points is performed to obtain the marker point ID tracking results; Key point detection is performed on the visible light image to obtain key point information, which includes the coordinates and ID of the key points; The motion attitude of the target in the visible light image is estimated to obtain the first pose data; The marker points are reprojected onto the visible light image and compared with the IDs of the key points to verify the accuracy of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking. For image frames that have lost marker information, the first pose data is used to fill in the missing information to obtain the motion data of the target being measured.

9. The motion capture system according to claim 4, wherein, The motion capture system further includes a second image processing unit connected to the visible light image sensor, used to perform one or more of the following processes on the visible light image: feature extraction, feature recognition, feature spatial positioning calculation, motion data calculation, and send the processing results to the processing unit.

10. The motion capture system according to claim 9, wherein, The second image processing unit is specifically used for: Key point detection is performed on the visible light image to obtain key point information, which includes the coordinates and ID of the key points; The motion attitude of the target in the visible light image is estimated to obtain the first pose data.

11. The motion capture system according to claim 10, wherein, The processing unit is configured to perform at least one of the following: Based on the coordinates of the marker point, a three-dimensional reconstruction of the marker point is performed; Perform ID tracking on the marker points to obtain the marker point ID tracking results; The marker points are reprojected onto the visible light image and compared with the IDs of the key points to verify the accuracy of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking. For image frames that have lost marker information, the first pose data is used to fill in the missing information to obtain the motion data of the target being measured.

12. The motion capture system according to claim 1, wherein, The motion capture module includes: At least one inertial sensor is disposed on the target under test and is used to measure the inertial data during the motion of the target under test and output the inertial data to the processing unit.

13. The motion capture system according to claim 12, wherein, The motion capture module also includes: At least one non-visible light image sensor, said non-visible light image sensor being used to sense the target under test in the near-infrared or infrared band and output an image in the near-infrared or infrared band; and / or, At least one visible light image sensor, the visible light image sensor being used to sense the target under test in the visible light band and output a visible light image.

14. The motion capture system according to claim 13, wherein, In the case where the motion capture system includes at least one non-visible light image sensor, the motion capture system also includes at least one near-infrared or infrared band supplementary light source for illuminating the target under test; At least one marker is arranged on the target being measured. Under the illumination of the supplementary light source, the marker has a contrast with the background in the imaging of the non-visible light image sensor.

15. The motion capture system according to claim 14, wherein, The motion capture system further includes: a first image processing unit, connected to the non-visible light image sensor, for tracking and spatially locating the marker points in the near-infrared or infrared band image, and outputting the marker point coordinates to the processing unit.

16. The motion capture system according to claim 15, wherein, In the case where the motion capture system includes at least one visible light image sensor, the motion capture system further includes: a second image processing unit, connected to the visible light image sensor, for: Key point detection is performed on the visible light image to obtain key point information, which includes the coordinates and ID of the key points; Motion attitude estimation is performed on the target in the visible light image to obtain the first pose data of the target.

17. The motion capture system according to claim 16, wherein, The processing unit is configured to perform at least one of the following: Based on the coordinates of the marker point, a three-dimensional reconstruction of the marker point is performed; The ID tracking of the marker points is performed to obtain the marker point ID tracking results; Calculate the second pose data of the target based on the coordinates of at least three non-collinear marker points on the target. Based on the inertial data, the motion attitude of the target under test is estimated to obtain the third pose data of the target under test. The marker points are reprojected onto the visible light image and compared with the IDs of the key points to verify the accuracy of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking. The motion data of the target under test are obtained by fusing the first pose data, the second pose data and the third pose data.

18. The motion capture system according to claim 1, 2, or 13, wherein, The motion capture module also includes an optical lens disposed in front of each of the sensors, which are used to sense the target being measured through the optical lens and obtain sensing data of different modalities.

19. A motion capture method, based on a motion capture system as described in any one of claims 1 to 18, comprising: The target under test is perceived to obtain perception data in different modalities; The sensing data of the different modalities are processed to obtain the motion data of the target being measured.

20. The motion capture method according to claim 19, wherein, The process of sensing the target and obtaining sensing data in different modalities includes: The target under test is sensed in the near-infrared or infrared band, and an image in the near-infrared or infrared band is output, wherein the image in the near-infrared or infrared band includes at least one marker point. The coordinates of the marker points in the near-infrared or infrared band image are obtained by tracking and spatial positioning calculation. The target under test is sensed in the visible light band, and a visible light image is output; Key point detection is performed on the visible light image to obtain key point information, which includes the coordinates and ID of the key points; The motion attitude of the target in the visible light image is estimated to obtain the first pose data; The coordinates of the key points and the coordinates of the marker points are matched to determine the ID of the marker points, thus obtaining the ID of the marker points.

21. The motion capture method according to claim 20, wherein, The process of processing the sensing data of the different modalities to obtain the motion data of the target under test includes: Based on the coordinates and ID of the marker point, a 3D reconstruction is performed on the marker point to obtain its 3D coordinates; Based on the three-dimensional coordinates of at least three non-collinear marker points, the pose of the target under test is calculated to obtain the second pose data; The motion data of the target under test are obtained by fusing the first pose data and the second pose data.

22. The motion capture method according to claim 19, wherein, The process of sensing the target and obtaining sensing data in different modalities includes: The target under test is sensed in the near-infrared or infrared band, and an image in the near-infrared or infrared band is output, wherein the image in the near-infrared or infrared band includes at least one marker point. The coordinates of the marker points in the near-infrared or infrared band image are obtained by tracking and spatial positioning calculation. The target under test is sensed in the visible light band, and a visible light image is output.

23. The motion capture method according to claim 22, wherein, The process of processing the sensing data of the different modalities to obtain the motion data of the target under test includes: determining the motion data of the target under test based on the coordinates of the marker points and the visible light image; The process of determining the motion data of the target based on the coordinates of the marker points and the visible light image includes: Based on the coordinates of the marker point, a three-dimensional reconstruction of the marker point is performed; The ID tracking of the marker points is performed to obtain the marker point ID tracking results; Key point detection is performed on the visible light image to obtain key point information, which includes the coordinates and ID of the key points; The motion attitude of the target in the visible light image is estimated to obtain the first pose data; The marker points are reprojected onto the visible light image and compared with the IDs of the key points to verify the accuracy of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking. For image frames that have lost marker information, the first pose data is used to fill in the missing information to obtain the motion data of the target being measured.

24. The motion capture method according to claim 19, wherein, The process of sensing the target and obtaining sensing data in different modalities includes: Measure the inertial data of the target object during its motion.

25. The motion capture method according to claim 24, wherein, The process of sensing the target and obtaining sensing data in different modalities also includes: The target is sensed in the near-infrared or infrared band, and an image in the near-infrared or infrared band is output, wherein the image in the near-infrared or infrared band includes at least one marker point; and / or, The target under test is sensed in the visible light band, and a visible light image is output.

26. The motion capture method according to claim 25, wherein, The process of processing the sensing data of the different modalities to obtain the motion data of the target under test includes: The coordinates of the marker points in the near-infrared or infrared band image are obtained by tracking and spatial positioning calculation. Based on the coordinates of the marker point, a three-dimensional reconstruction of the marker point is performed; The ID tracking of the marker points is performed to obtain the marker point ID tracking results; The first pose data of the target is calculated based on the coordinates of at least three non-collinear marker points on the target. Key point detection is performed on the visible light image to obtain key point information, which includes the coordinates and ID of the key points; Motion pose estimation is performed on the target in the visible light image to obtain the second pose data of the target; Based on the inertial data, the motion attitude of the target under test is estimated to obtain the third pose data of the target under test. The marker points are reprojected onto the visible light image and compared with the IDs of the key points to verify the accuracy of the marker point ID tracking results or to identify the IDs of marker points that have lost tracking. The motion data of the target under test are obtained by fusing the first pose data, the second pose data and the third pose data.

27. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the motion capture method as described in any one of claims 19 to 26.

28. A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the motion capture method as described in any one of claims 19 to 26.

29. A chip comprising a processor and a communication interface coupled to the processor, the processor being configured to run a program or instructions to implement the motion capture method as described in any one of claims 19 to 26.

30. A computer program product comprising a computer program that, when executed by a processor, implements the motion capture method as described in any one of claims 19 to 26.

Images