Method and apparatus for determining relative positions of multiple entities, and device and medium

Abstract

A method and apparatus for determining relative positions of multiple entities, and a device and a medium. The method for determining relative positions of multiple entities comprises: by means of a visual sensor provided for each entity from among a plurality of entities, acquiring image data of the environment where the corresponding entity is located (S101); for any entity, performing feature point extraction on image data corresponding to the entity, so as to obtain a plurality of feature points corresponding to the entity (S102); on the basis of the plurality of feature points corresponding to the entity, constructing a sparse map for the entity in a corresponding entity coordinate system (S103); and performing alignment on sparse maps corresponding to any two of the plurality of entities, so as to determine a relative position between any two entities (S104). By means of constructing a sparse map for each entity from among a plurality of entities and aligning the sparse maps, the relative positions among the plurality of entities can be automatically determined, without relying on an external calibration object, thereby achieving a strong adaptability and a high flexibility, and thus reducing the cost and time consumed for adaptation, and improving the robustness, accuracy and reliability of the determination of relative positions.

Description

Methods, apparatus, equipment and media for determining the relative positions of multiple entities

[0001] Cross-references to related applications

[0002] This disclosure claims priority to the Chinese Patent Publication No. 2025101219230 filed on January 26, 2025, entitled "Method, Apparatus, Equipment and Medium for Determining the Relative Position of Multiple Entities". Technical Field

[0003] This disclosure relates to the field of positioning technology, specifically to a method, apparatus, device, and medium for determining the relative position of multiple entities. Background Technology

[0004] In multi-entity collaboration scenarios, whether it's robot collaborative operations, autonomous vehicle platooning, or the interaction of multiple terminal devices in complex environments, determining the relative positions of these entities is crucial for the efficiency and safety of multi-entity collaboration. Summary of the Invention

[0005] This disclosure aims to at least partially address one of the technical problems in the related art.

[0006] Therefore, the first objective of this disclosure is to propose a method for determining the relative positions of multiple entities.

[0007] The second objective of this disclosure is to provide a device for determining the relative positions of multiple entities.

[0008] The third objective of this disclosure is to propose an electronic device.

[0009] The fourth objective of this disclosure is to provide a computer-readable storage medium.

[0010] The fifth objective of this disclosure is to provide a computer program product.

[0011] To achieve the above objectives, the first aspect of this disclosure proposes a method for determining the relative position of multiple entities, comprising: acquiring image data of the environment in which the corresponding entity is located through a visual sensor equipped with each of the multiple entities; extracting feature points from the image data corresponding to any entity to obtain multiple feature points corresponding to the entity; constructing a sparse map of the entity in the corresponding entity coordinate system based on the multiple feature points corresponding to the entity; and aligning the sparse maps corresponding to any two entities among the multiple entities to determine the relative position between the two entities.

[0012] To achieve the above objectives, a second aspect of this disclosure provides a multi-entity relative position determination device, comprising: an acquisition module, configured to acquire image data of the environment in which the corresponding entity is located through a visual sensor equipped with each of the multiple entities; an extraction module, configured to extract feature points from the image data corresponding to any one of the entities to obtain multiple feature points corresponding to the entity; a construction module, configured to construct a sparse map of the entities based on the multiple feature points corresponding to the entities; and an alignment module, configured to align the sparse maps corresponding to any two of the multiple entities to determine the relative position between the two entities.

[0013] To achieve the above objectives, a third aspect of this disclosure provides an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect of the above disclosure.

[0014] To achieve the above objectives, a fourth aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method described in the first aspect of the above disclosure.

[0015] To achieve the above objectives, a fifth aspect of this disclosure provides a computer program product including a computer program that, when executed by a processor, implements the method described in the first aspect of the above embodiment.

[0016] The method, apparatus, device, and medium for determining the relative position of multiple entities disclosed herein can automatically determine the relative position between multiple entities by constructing a sparse map for each entity and aligning the sparse maps. This does not rely on external calibration objects, is applicable to various complex and dynamic environments, and is highly adaptable and flexible. Furthermore, since it does not require pre-defining the pattern or features of calibration objects, it reduces the complexity of algorithm development and adaptation, thereby reducing adaptation costs and time. Even when some feature points are occluded or the environment changes, it can still perform sparse map alignment and positioning using the remaining feature points, improving the robustness, accuracy, and reliability of relative position determination.

[0017] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0019] Figure 1 is a flowchart illustrating a method for determining the relative positions of multiple entities according to an embodiment of this disclosure;

[0020] Figure 2 is a flowchart illustrating a method for determining the relative positions of multiple entities according to another embodiment of this disclosure;

[0021] Figure 3 is a flowchart illustrating a method for determining the relative positions of multiple entities according to another embodiment of this disclosure;

[0022] Figure 4 is a flowchart illustrating a method for determining the relative positions of multiple entities provided in another embodiment of this disclosure;

[0023] Figure 5 is a flowchart illustrating a method for determining the relative positions of multiple entities provided in another embodiment of this disclosure;

[0024] Figure 6 is a schematic diagram of the structure of a multi-entity relative position determination device provided in another embodiment of this disclosure. Detailed Implementation

[0025] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0026] In related technologies, determining the relative positions between multiple entities typically relies on external markers (such as QR codes with prior information) or calibration boards. This involves multiple entities equipped with visual sensors observing the same characteristic object to determine the relative positions between them. Specifically, this may include the following steps:

[0027] 1. External calibration object: An object with known characteristics is placed in the working environment. This object can be a QR code or a calibration board, etc. These objects usually have feature points that are easy to identify and calculate, such as the positioning marks of a QR code or a specific pattern on a calibration board.

[0028] 2. Relative position calculation: The relative position between each entity and the calibration object can be calculated, and the calibration algorithm is used to determine the positional relationship between the calibration object and the corresponding entity. Then, after obtaining the positional relationship between each entity and the calibration object, the relative position between multiple entities can be indirectly derived by using the position of the calibration object as an intermediary.

[0029] However, this method has the following drawbacks:

[0030] 1. Reliance on external markers: Marker objects need to be placed in the environment, which limits the flexibility of use cases.

[0031] 2. The type of marker needs to be predefined: Taking the calibration plate as an example, the pattern drawn on the calibration plate needs to be precisely predefined, and the algorithm needs to be adapted to this definition, which results in high algorithm adaptation costs.

[0032] 3. Poor reliability: The reliability of the system will be affected when the environment changes or the calibration object is blocked.

[0033] To address at least one of the aforementioned problems, this disclosure proposes a method, apparatus, device, and medium for determining the relative positions of multiple entities.

[0034] The following description, with reference to the accompanying drawings, outlines a method, apparatus, device, and medium for determining the relative positions of multiple entities according to embodiments of this disclosure.

[0035] Figure 1 is a flowchart illustrating a method for determining the relative positions of multiple entities according to an embodiment of this disclosure.

[0036] This disclosure illustrates by exemplifying the multi-entity relative position determination method configured in a multi-entity relative position determination device. This multi-entity relative position determination device can be applied to any electronic device so that the electronic device can perform the multi-entity relative position determination function.

[0037] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), industrial computers, host computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0038] As shown in Figure 1, the method includes the following steps:

[0039] Step S101: Obtain image data of the environment in which the corresponding entity is located through the visual sensor equipped in each of the multiple entities.

[0040] It should be noted that this disclosure does not impose excessive restrictions on entities. For example, in a scenario of robot collaborative operation, the entity can be a robot; in a scenario of autonomous vehicles platooning, the entity can be a vehicle; in a scenario of interaction among multiple terminal devices, the entity can be a terminal device, and so on.

[0041] The visual sensor can be a monocular camera, a binocular camera, an RGB-D (Red-Green-Blue-Depth) camera, etc., and this disclosure does not limit it.

[0042] In this embodiment of the disclosure, each of the multiple entities is equipped with a corresponding visual sensor, thereby enabling the acquisition of image data of the environment in which the corresponding entity is located through the visual sensor equipped with each of the multiple entities.

[0043] As one possible implementation, image data can include multiple consecutive image frames.

[0044] Step S102: For any entity, feature points are extracted from the image data corresponding to the entity to obtain multiple feature points corresponding to the entity.

[0045] The feature points may include corner points, edge points, spots, etc., and this disclosure does not limit them.

[0046] In this embodiment of the disclosure, for any entity, feature points can be extracted from the image data corresponding to the entity to obtain multiple feature points corresponding to the entity.

[0047] Optionally, feature point detection algorithms can be used to detect or identify feature points in the image data, and the detected or identified feature points can be described to obtain corresponding feature point descriptors. For example, SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), and ORB (Oriented Fast and Rotated BRIEF) algorithms can be used to detect feature points in the image data, and the detected feature points can be described to obtain their corresponding feature point descriptors.

[0048] Alternatively, before extracting feature points from the image data corresponding to the entity, the image data can be preprocessed. Preprocessing may include denoising, grayscale conversion, etc., and this disclosure does not impose any limitations on this. This facilitates subsequent processing of the image data and improves the accuracy and effectiveness of feature point extraction.

[0049] Step S103: Construct a sparse map of the entity based on multiple feature points corresponding to the entity.

[0050] In this embodiment of the disclosure, multiple feature points corresponding to an entity can be used to construct a sparse map of the entity.

[0051] It should be noted that each entity has a corresponding entity coordinate system, which can be, for example, the world coordinate system of the corresponding entity.

[0052] As an example, the spatial coordinates of each feature point corresponding to an entity in the corresponding entity coordinate system can be obtained. Based on the spatial coordinates of each feature point corresponding to an entity in the corresponding entity coordinate system, the feature points corresponding to the entity can be fused to obtain a sparse map of the entity.

[0053] Step S104: Align the sparse maps corresponding to any two entities among the multiple entities to determine the relative positions between the two entities.

[0054] The relative position can include, for example, six degrees of freedom data, which can include position in space, rotation angle, and relative distance. For instance, in three-dimensional space, with a coordinate system of O-XYZ, the relative position can include the position x along the X-axis, the position y along the Y-axis, the position z along the Z-axis, as well as the rotation angles around the Z-axis (roll, pitch, and yaw) and the relative distance d.

[0055] The multi-entity relative position determination method of this disclosure acquires image data of the environment in which each entity is located through a visual sensor equipped with the entity. For any entity, feature points are extracted from the image data corresponding to the entity to obtain multiple feature points corresponding to the entity. Based on the multiple feature points corresponding to the entity, a sparse map of the entity is constructed. The sparse maps corresponding to any two entities are aligned to determine the relative position between the two entities. Thus, by constructing and aligning the sparse maps of each entity in the multiple entities, the relative position between multiple entities can be automatically determined without relying on external calibration objects. It is applicable to various complex and dynamic environments, with strong adaptability and high flexibility. Since it does not require pre-defining the pattern or features of the calibration object, it reduces the complexity of algorithm development and adaptation, thereby reducing adaptation costs and time. Even when some feature points are occluded or the environment changes, sparse map alignment and positioning can still be performed using the remaining feature points, improving the robustness, accuracy, and reliability of relative position determination.

[0056] In the case where any two entities include a first entity and a second entity, in order to clearly illustrate how the sparse map corresponding to any two entities among multiple entities is aligned in the above embodiments of this disclosure to determine the relative position between any two entities, this disclosure also proposes a method for determining the relative position of multiple entities.

[0057] Figure 2 is a flowchart illustrating a method for determining the relative positions of multiple entities provided in another embodiment of this disclosure.

[0058] As shown in Figure 2, the method includes the following steps:

[0059] Step S201: Obtain image data of the environment in which the corresponding entity is located through the visual sensor equipped in each of the multiple entities.

[0060] Step S202: For any entity, feature points are extracted from the image data corresponding to the entity to obtain multiple feature points corresponding to the entity.

[0061] Step S203: Construct a sparse map of the entity based on multiple feature points corresponding to the entity.

[0062] It should be noted that the execution process of steps S201 to S203 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0063] Step S204: Obtain the similarity between the sparse map of the first entity and the sparse map of the second entity among any two entities.

[0064] As one possible implementation, as shown in Figure 3, the following steps can be used to determine the similarity between the sparse maps of the first entity and the second entity for any two entities:

[0065] Step S2041: Determine the first number of feature points in the sparse map of the first entity that match the feature points in the sparse map of the second entity.

[0066] In this embodiment of the disclosure, for any feature point in the sparse map of the first entity, the feature point can be matched with each feature point in the sparse map of the second entity. When there is a first feature point in the sparse map of the second entity that matches the feature point, it is determined that the feature point matches the feature point in the sparse map of the second entity. Thus, the feature points in the sparse map of the first entity that match the feature points in the sparse map of the second entity are accumulated to obtain a first number.

[0067] When matching any feature point in the sparse map of the first entity with each feature point in the sparse map of the second entity, matching algorithms such as FLANN (Fast Library for Approximate Nearest Neighbors) and BFMatcher (Brute-Force Matcher) can be used.

[0068] In order to accurately match any feature point in the sparse map of the first entity with each feature point in the sparse map of the second entity, in one possible implementation of this disclosure, for any feature point in the sparse map of the first entity, the distance between the feature point descriptor of the feature point and the feature point descriptor of any feature point in the sparse map of the second entity can be determined; when there is a second feature point in the sparse map of the second entity, and the distance between the feature point descriptor of the second feature point and the feature point descriptor of the first feature point is less than a second set threshold, then the second feature point is determined to be the first feature point that matches the first feature point.

[0069] It should be noted that, for any feature point in the sparse map of the first entity, when determining the distance between the feature point descriptor of that feature point and the feature point descriptor of any feature point in the sparse map of the second entity, a distance metric method (such as Euclidean distance, Manhattan distance, cosine similarity, etc.) can be used.

[0070] The second set threshold can be preset, and this disclosure does not restrict the value of the second set threshold.

[0071] Step S2042: Determine the first coefficient based on the first quantity and the total number of feature points in the sparse map of the first entity.

[0072] As an example, the first coefficient can be determined as the ratio of the first quantity to the total number of feature points in the sparse map of the first entity.

[0073] For example, suppose the total number of feature points in the sparse map of the first entity is a1, and the first number of feature points in the sparse map of the first entity that match the feature points in the sparse map of the second entity is b1, then the first coefficient is (a1 / b1).

[0074] Step S2043: Determine the second number of feature points in the sparse map of the second entity that match the feature points in the sparse map of the first entity.

[0075] It should be noted that the method for determining the second quantity is similar to the method for determining the first quantity, and will not be repeated here.

[0076] It should also be noted that the first quantity may be the same as or different from the second quantity, and this disclosure does not impose any restrictions on this.

[0077] Step S2044: Determine the second coefficient based on the second quantity and the total number of feature points in the sparse map of the second entity.

[0078] As an example, the second coefficient can be determined as the ratio of the second quantity to the total number of feature points in the sparse map of the second entity.

[0079] For example, suppose the total number of feature points in the sparse map of the second entity is a2, and the second number of feature points in the sparse map of the second entity that match the feature points in the sparse map of the first entity is b2, then the second coefficient is (a2 / b2).

[0080] It should be noted that this disclosure does not restrict the execution order of steps S2041-S2042 and steps S2043-S2044. This disclosure only exemplifies that steps S2041-S2042 are executed before steps S2043-S2044. Steps S2041-S2042 can also be executed in parallel with steps S2043-S2044, or steps S2041-S2042 can also be executed after steps S2043-S2044.

[0081] Step S2045: Based on the first coefficient and / or the second coefficient, determine the similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities.

[0082] As an example, the first coefficient can be determined as the similarity between the sparse maps of the first entity and the sparse maps of the second entity in any two entities.

[0083] As another example, the second coefficient can be determined as the similarity between the sparse maps of the first entity and the sparse maps of the second entity in any two entities.

[0084] As another example, a weighted average of the first and second coefficients can be taken to obtain the similarity between the sparse maps of the first entity and the sparse maps of the second entity for any two entities.

[0085] Therefore, the similarity between sparse maps of any two entities can be determined effectively and accurately.

[0086] Step S205: In response to the similarity being greater than a first set threshold, the sparse map of the first entity and the sparse map of the second entity are aligned to obtain the first relative pose between the first entity and the second entity.

[0087] The first set threshold can be preset, and this disclosure does not restrict the value of the first set threshold.

[0088] The first relative pose may include a translation vector and a rotation matrix.

[0089] In this embodiment of the disclosure, when the similarity is greater than a first set threshold, it indicates that the first entity and the second entity have observed similar scenes and can form a loop. At this time, the sparse map of the first entity and the sparse map of the second entity can be aligned to obtain the first relative pose between the first entity and the second entity.

[0090] As one possible implementation, feature points in the sparse map of the first entity and feature points in the sparse map of the second entity can be matched to obtain a first pair of matched feature points; based on the first pair of matched feature points, the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity can be determined; based on the transformation relationship, the first relative pose between the first entity and the second entity can be determined.

[0091] It should be noted that this disclosure does not limit the number of first feature point pairs.

[0092] In this embodiment of the disclosure, each feature point in the sparse map of the first entity and each feature point in the sparse map of the second entity can be matched to obtain a first pair of matched feature points. For example, a feature matching algorithm (such as FLANN, BFMatcher, etc.) can be used to match each feature point in the sparse map of the first entity and each feature point in the sparse map of the second entity to obtain a first pair of matched feature points.

[0093] In this embodiment of the disclosure, the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity can be determined based on the matched first feature point pair. As an example, the spatial coordinates of each feature point in the matched first feature point pair in the corresponding entity coordinate system can be obtained, and then one of the least squares method, the Iterative Closest Point (ICP) algorithm, etc., can be used to determine the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity.

[0094] In this embodiment of the disclosure, the first relative pose between the first entity and the second entity can be determined based on the transformation relationship.

[0095] As an example, assume that the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity is as shown in the following formula: P1=RP2+T; (1)

[0096] Where P1 represents the spatial coordinates of the feature point of the first entity in the entity coordinate system of the first entity, and is a 3x1 vector; P2 represents the spatial coordinates of the feature point of the second entity in the entity coordinate system of the second entity, and is a 3x1 vector; R is a 3x3 rotation matrix, representing the rotation operation from the entity coordinate system of the second entity to the entity coordinate system of the first entity; T is a 3x1 translation vector, representing the translation operation from the entity coordinate system of the second entity to the entity coordinate system of the first entity.

[0097] Furthermore, according to formula (1), the first relative pose between the first entity and the second entity can be determined as follows: the rotation matrix of the first entity relative to the second entity is R, and the translation vector is T.

[0098] As another example, suppose the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity is as shown in the following formula: P1=R′P2+T′; (2)

[0099] Where P1 represents the spatial coordinates of the feature points of the first entity in the entity coordinate system of the first entity, and is a 3x1 vector; P2 represents the spatial coordinates of the feature points of the second entity in the entity coordinate system of the second entity, and is a 3x1 vector; R' is a 3x3 rotation matrix, representing the rotation operation from the entity coordinate system of the second entity to the entity coordinate system of the first entity; T' is a 3x1 translation vector, representing the translation operation from the entity coordinate system of the second entity to the entity coordinate system of the first entity.

[0100] Furthermore, according to formula (2), the first relative pose between the first entity and the second entity can be determined as follows: the rotation matrix of the second entity relative to the first entity is R', and the translation vector is T'.

[0101] Therefore, by finding matching feature point pairs in the sparse map of any two entities, the transformation relationship between the entity coordinate systems of any two entities can be effectively determined. Thus, based on the transformation relationship, the first relative pose between the first entity and the second entity can be effectively and accurately determined.

[0102] To improve the accuracy of the feature point matching process, in one possible implementation of this disclosure, after obtaining the first matched feature point pair, the RANSAC (Random Sample Consensus) algorithm can be used to perform geometric consistency verification on the obtained first matched feature point pair. This can effectively remove erroneous matches in the feature point matching process and improve the accuracy and robustness of subsequent tasks.

[0103] It is understandable that there may be cases where the similarity is not greater than the first set threshold. In this case, it indicates that the first entity and the second entity have not observed similar scenes. Therefore, in one possible implementation of this disclosure embodiment, when the similarity is not greater than the first set threshold, it can be determined from multiple entities whether a target entity exists. Wherein, the similarity between the sparse map of the target entity and the sparse map of the first entity is greater than the first set threshold, and the similarity between the sparse map of the target entity and the sparse map of the second entity is greater than the first set threshold. When a target entity exists among multiple entities, the first relative pose between the first entity and the second entity can be determined based on the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the target entity, and the transformation relationship between the entity coordinate system of the second entity and the entity coordinate system of the target entity.

[0104] It should be noted that the methods for obtaining the similarity between the sparse map of the target entity and the sparse map of the first entity, and the methods for obtaining the similarity between the sparse map of the target entity and the sparse map of the second entity, are similar to the methods for obtaining the similarity between the sparse map of the first entity and the sparse map of the second entity in step S204, and will not be elaborated here.

[0105] In this embodiment of the disclosure, when a target entity exists among multiple entities, the first relative pose between the first entity and the second entity can be determined based on the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the target entity, and the transformation relationship between the entity coordinate system of the second entity and the entity coordinate system of the target entity.

[0106] As an example, when a target entity exists among multiple entities, assume that the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the target entity is as shown in the following formula: P = RP1 + T; (3)

[0107] Where P1 represents the spatial coordinates of the feature point of the first entity in the entity coordinate system of the first entity, which is a 3x1 vector; P represents the spatial coordinates of the feature point of the target entity in the entity coordinate system of the target entity, which is a 3x1 vector; R is a 3x3 rotation matrix, representing the rotation operation from the entity coordinate system of the target entity to the entity coordinate system of the first entity; T is a 3x1 translation vector, representing the translation operation from the entity coordinate system of the target entity to the entity coordinate system of the first entity.

[0108] The transformation relationship between the entity coordinate system of the second entity and the entity coordinate system of the target entity is shown in the following formula: P2=R′P+T′; (4)

[0109] Where P2 represents the spatial coordinates of the feature points of the second entity in the entity coordinate system of the second entity, which is a 3x1 vector; P represents the spatial coordinates of the feature points of the target entity in the entity coordinate system of the target entity, which is a 3x1 vector; R' is a 3x3 rotation matrix, representing the rotation operation from the entity coordinate system of the target entity to the entity coordinate system of the first entity; T' is a 3x1 translation vector, representing the translation operation from the entity coordinate system of the target entity to the entity coordinate system of the first entity.

[0110] Based on the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the target entity, and the transformation relationship between the entity coordinate system of the second entity and the entity coordinate system of the target entity, the first relative pose between the first entity and the second entity can be determined according to the following formula: P2=R′(RP1+T)+T′=R′RP1+(R′T+T′); (5)

[0111] The first relative pose between the first entity and the second entity is: the rotation matrix of the second entity relative to the first entity is R'R, and the translation vector is (R'T+T').

[0112] Therefore, if the similarity between the sparse map of the first entity and the sparse map of the second entity is not greater than the first set threshold, the first relative pose between the first entity and the second entity can be effectively and accurately determined by means of the target entity.

[0113] Step S206: Analyze the first relative pose between the first entity and the second entity to obtain the relative position between the first entity and the second entity.

[0114] It should be noted that the explanation of the relative position in step S104 also applies to this embodiment, and will not be repeated here.

[0115] Specifically, when the first relative pose includes a translation vector and a rotation matrix, the spatial position of the first entity in the entity coordinate system of the second entity and / or the spatial position of the first entity in the entity coordinate system of the second entity, as well as the relative distance between the first entity and the second entity, can be obtained based on the translation vector in the first relative pose between the first entity and the second entity; the rotation angle of the first entity around the corresponding coordinate axis of the entity coordinate system of the second entity can be obtained based on the rotation matrix in the first relative pose between the first entity and the second entity.

[0116] The multi-entity relative position determination method of this disclosure obtains the similarity between the sparse maps of a first entity and a second entity in any two entities; in response to the similarity being greater than a first preset threshold, the sparse maps of the first entity and the second entity are aligned to obtain a first relative pose between the first entity and the second entity; the first relative pose between the first entity and the second entity is parsed to obtain the relative position between the first entity and the second entity. Therefore, by calculating the similarity between the sparse maps of any two entities and aligning the sparse maps when the similarity meets certain conditions, the relative position between entities is determined, improving the accuracy and efficiency of determining the relative position.

[0117] In the case that the image data corresponding to each entity includes multiple consecutive image frames, and each image frame has multiple corresponding feature points, in order to clearly illustrate how a sparse map of an entity is constructed based on multiple feature points corresponding to the entity in the above embodiments of this disclosure, this disclosure also proposes a method for determining the relative position of multiple entities.

[0118] Figure 4 is a flowchart illustrating a method for determining the relative positions of multiple entities provided in another embodiment of this disclosure.

[0119] As shown in Figure 4, the method includes the following steps:

[0120] Step S401: Obtain image data of the environment in which the corresponding entity is located through the visual sensor equipped in each of the multiple entities.

[0121] Step S402: For any entity, feature points are extracted from the image data corresponding to the entity to obtain multiple feature points corresponding to the entity.

[0122] It should be noted that the execution process of steps S401 to S402 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0123] Step S403: Sort the images of the entity according to the order of their acquisition times to obtain the sorted sequence of the entity's images.

[0124] Step S404: Determine the camera coordinate system of the first image frame in the image sorting sequence acquired by the vision sensor of the entity as the entity coordinate system.

[0125] In other words, the camera coordinate system of the first image frame in the sequence of images acquired by the visual sensor of the entity coincides with the entity's entity coordinate system.

[0126] Step S405: According to the image sorting sequence of the entity, determine the spatial coordinates of each feature point corresponding to each image frame in the entity's entity coordinate system.

[0127] Optionally, as shown in Figure 5, the following steps can be used to determine the spatial coordinates of each feature point corresponding to each image frame in the entity's entity coordinate system:

[0128] Step S4051: For any feature point in the image frame with sequence number 1 in the image sorting sequence of the entity, determine the spatial coordinates of the feature point in the corresponding entity coordinate system based on the image coordinates of the feature point.

[0129] Image coordinates can be used to indicate the coordinate position of a corresponding feature point in a corresponding image frame.

[0130] In this embodiment of the disclosure, the spatial coordinates of the feature points in the corresponding entity coordinate system can be determined based on the image coordinates of the feature points. Specifically, feature point triangulation can be performed on the feature points based on the image coordinates of the feature points and the intrinsic parameters of the vision sensor equipped on the entity (such as focal length, principal point coordinates, etc.) to obtain the spatial coordinates of the feature points in the corresponding entity coordinate system.

[0131] Step S4052: For the (i+1)th image frame in the image sorting sequence corresponding to the entity, match multiple feature points of the i-th image frame with multiple feature points of the (i+1)th image frame to obtain a matched second feature point pair.

[0132] Where i can be a positive integer that is not zero.

[0133] It should be noted that this disclosure does not limit the number of second feature point pairs.

[0134] As an example, for the (i+1)th image frame in the image sorting sequence corresponding to an entity, matching algorithms such as FLANN and BFMatcher can be used to match multiple feature points of the i-th image frame with multiple feature points of the (i+1)-th image frame to obtain a matching second feature point pair.

[0135] During feature point matching, erroneous matching pairs often occur due to factors such as image noise, illumination variations, and viewpoint differences. These erroneous matching pairs negatively impact subsequent data applications. To improve the accuracy of feature point matching, in one possible implementation of this disclosure, after obtaining the matched second feature point pair, the RANSAC (Random Sample Consensus) algorithm can be used to verify the geometric consistency of the obtained matched second feature point pair. This effectively removes erroneous matches during the feature point matching process, improving the accuracy and robustness of subsequent tasks.

[0136] Step S4053: Based on the matched second feature point pair, determine the second relative pose of the entity's visual sensor when acquiring the (i+1)th image frame relative to when acquiring the ith image frame.

[0137] The second relative pose may include a translation vector and a rotation matrix.

[0138] In this embodiment of the disclosure, the second relative pose of the visual sensor equipped with the entity when acquiring the (i+1)th image frame relative to the acquisition of the i-th image frame can be determined based on the matching second feature point pairs between the i-th and (i+1)-th image frames. Specifically, the second relative pose of the visual sensor equipped with the entity when acquiring the (i+1)-th image frame relative to the acquisition of the i-th image frame can be determined based on the spatial coordinates of the feature points belonging to the i-th image frame and the image coordinates of the feature points belonging to the (i+1)-th image frame in the matching second feature point pairs between the i-th and (i+1)-th image frames. For example, the second relative pose of the visual sensor equipped with the entity when acquiring the (i+1)-th image frame relative to the acquisition of the i-th image frame can be determined using a pose estimation algorithm (e.g., the PnP (Perspective-n-Points) algorithm) based on the spatial coordinates of the feature points belonging to the i-th image frame and the image coordinates of the feature points belonging to the (i+1)-th image frame in the matching second feature point pairs between the i-th and (i+1)-th image frames.

[0139] Step S4054: For any feature point in the (i+1)th image frame, determine the spatial coordinates of the feature point in the corresponding entity coordinate system based on the second relative pose and the image coordinates of the feature point.

[0140] As an example, firstly, for any feature point in the (i+1)th image, based on the image coordinates of the feature point, determine the spatial coordinates of the feature point in the camera coordinate system when the vision sensor equipped with the entity acquires the (i+1)th image frame. For example, based on the image coordinates of the feature point and the intrinsic parameters of the vision sensor equipped with the entity (such as focal length, principal point coordinates, etc.), feature point triangulation can be performed on the feature point to obtain the spatial coordinates of the feature point in the camera coordinate system when the vision sensor equipped with the entity acquires the (i+1)th image frame. Secondly, based on the second relative pose of the vision sensor equipped with the entity when acquiring the (i+1)th image frame relative to the acquisition of the ith image frame, the estimated pose of the camera coordinate system equipped with the entity equipped with the vision sensor relative to the entity's entity coordinate system when acquiring the (i+1)th image frame can be determined. Finally, based on the estimated pose and the intrinsic parameters of the vision sensor equipped with the entity (such as focal length, principal point coordinates, etc.), the spatial coordinates of the feature point in the camera coordinate system when the vision sensor equipped with the entity acquires the (i+1)th image frame are converted into spatial coordinates in the entity coordinate system.

[0141] Specifically, when determining the estimated pose of the camera coordinate system relative to the entity's entity coordinate system at the time of acquiring the (i+1)th image frame, based on the second relative pose of the entity's visual sensor at the time of acquiring the (i+1)th image frame relative to the time of acquiring the (i)th image frame, for example, it can be determined based on the second relative pose of the entity's visual sensor at the time of acquiring the (i+1)th image frame relative to the time of acquiring the (i)th image frame, and the second relative pose corresponding to any adjacent image frame before the (i+1)th image frame.

[0142] For example, assuming i is 3, the second relative pose of the entity's vision sensor when acquiring the 4th image frame relative to when acquiring the 3rd image frame is: translation vector T 34 Rotation matrix R 34 The second relative pose of the visual sensor for any adjacent image frames before acquiring the fourth image frame includes: the second relative pose of the visual sensor when acquiring the third image frame relative to when acquiring the second image frame is: translation vector T 23 Rotation matrix R 23 The second relative pose of the vision sensor when acquiring the second image frame, relative to when acquiring the first image frame, is: translation vector T 12 Rotation matrix R 12 The estimated pose of the entity's camera coordinate system relative to the entity's entity coordinate system when the entity's vision sensor acquires the (i+1)th image frame is: rotation matrix (R... 34 R 23 R12 ), translation vector (R) 34 R 23 T 12 +R 34 T 23 +T 34 Therefore, the estimated pose of the camera coordinate system relative to the entity's entity coordinate system can be effectively determined when the vision sensor equipped on the entity acquires the (i+1)th image frame.

[0143] In summary, the spatial coordinates of each feature point in a given image frame can be determined frame by frame, based on the acquisition time of multiple consecutive image frames corresponding to the entity.

[0144] Step S406: Construct a sparse map of the entity based on the spatial coordinates of each feature point corresponding to the entity in the corresponding entity coordinate system.

[0145] In this embodiment of the disclosure, the feature points corresponding to the entity can be fused based on the spatial coordinates of each feature point in the corresponding entity coordinate system to obtain a sparse map of the entity.

[0146] Step S407: Align the sparse maps corresponding to any two entities among the multiple entities to determine the relative positions between the two entities.

[0147] It should be noted that the execution process of step S407 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0148] This disclosure discloses a method for determining the relative positions of multiple entities. It obtains an image sorting sequence of entities by arranging the images of multiple consecutive image frames corresponding to each entity in chronological order of acquisition. The camera coordinate system of the first image frame in the image sorting sequence acquired by the entity's visual sensor is then defined as the entity's entity coordinate system. Following the image sorting sequence, the spatial coordinates of each feature point corresponding to each image frame in the entity's entity coordinate system are determined sequentially. Based on the spatial coordinates of each feature point in its corresponding entity coordinate system, a sparse map of the entities is constructed. This effectively achieves the construction of a sparse map for each entity, providing crucial data support for subsequently determining the relative positions between multiple entities.

[0149] To clearly illustrate the method for determining the relative positions of multiple entities disclosed herein, a detailed description is provided below with examples.

[0150] As an example, a method for determining the relative positions of multiple entities may include the following steps:

[0151] 1. For any entity among multiple entities, perform real-time SLAM (Simultaneous Localization and Mapping) during entity runtime, specifically:

[0152] 1.1 Data Acquisition

[0153] The entity acquires image data of its environment using at least one vision sensor.

[0154] The visual sensor can be a monocular camera, a binocular camera, or an RGB-D camera, etc., and this disclosure does not limit it.

[0155] 1.2 Feature Point Extraction

[0156] Feature point extraction is performed on image data to obtain multiple feature points corresponding to entities. For example, algorithms such as SIFT, SURF, and ORB can be used to detect or recognize feature points in image data, and the detected or recognized feature points can be described to obtain feature point descriptors.

[0157] Among them, feature points can be, but are not limited to, corner points, edges, etc.

[0158] 1.3 Feature Point Matching

[0159] When the image data includes multiple consecutive image frames, feature point matching can be performed between adjacent image frames in the multiple consecutive image frames to obtain matching second feature point pairs. For example, matching algorithms such as FLANN and BFMatcher can be used to obtain matching second feature point pairs between adjacent image frames.

[0160] It should be noted that erroneous matches may occur during feature point matching. To improve the accuracy of matching, the RANSAC (Random Sample Consensus) algorithm can be used to perform geometric consistency verification on the obtained second feature point pairs to remove erroneous matches.

[0161] 1.4 Pose Estimation

[0162] Based on the second feature point pairs matched between adjacent image frames, a pose estimation algorithm (such as the PnP algorithm) is used to estimate the camera pose, obtaining the second relative pose of the visual sensor when acquiring corresponding adjacent image frames. Furthermore, by utilizing multi-view set relationships, the estimated pose of the entity's visual sensor when acquiring different image frames is calculated.

[0163] 1.5 Sparse Map Construction

[0164] Based on the estimated pose, the feature points are back-projected to obtain their spatial coordinates in 3D space. Then, based on the spatial coordinates of each feature point corresponding to the entity, the feature points corresponding to the entity are fused to obtain a sparse map of the entity.

[0165] Optionally, a bundle adjustment (BA) algorithm or filtering method (such as EKF (Extended Kalman Filter) or UKF (Unscented Kalman Filter)) can be used to globally optimize the estimated pose and spatial coordinates of feature points in order to reduce errors and improve map accuracy.

[0166] 2. Align the sparse map of any two entities to determine their relative positions. Specifically:

[0167] 2.1 Feature Point Matching

[0168] Feature point matching is performed on the feature points of the sparse map of any two entities to obtain the first matching feature point pair. For example, matching algorithms such as FLANN and BFMatcher can be used to obtain the first matching feature point pair.

[0169] It should be noted that in step 1.2, after extracting the feature points of the entity, the feature descriptors of the entity's feature points can be saved for subsequent data applications, such as feature point matching.

[0170] 2.2 Determining the relative position between any two entities

[0171] The similarity between the sparse maps of any two entities is determined; when the similarity is greater than a first set threshold, it is determined that the visual sensors of the corresponding two entities have observed similar scenes, which can form a loop. Thus, based on the first feature point pair matched between the two entities, the transformation relationship between the world coordinate system (denoted as entity coordinate system in this disclosure) of the two entities can be determined, and based on the transformation relationship, the relative position between the two entities can be determined.

[0172] Among them, based on the transformation relationship, the coordinates of one entity in any two entities can be transformed to the world coordinate system of the other entity, thus realizing the alignment of sparse maps of the two entities.

[0173] Optionally, the aligned sparse maps can be merged to obtain a merged map of the corresponding two entities, and so on, to achieve map fusion between multiple entities, which will not be elaborated here.

[0174] Understandably, as the SLAM algorithms for each entity continue to run, the relative positions between entities can be updated and dynamically adjusted in real time. Furthermore, the continuous operation of the SLAM process and map fusion algorithm ensures that the complete merged map is synchronously updated for each continuously running entity, and new map areas explored by each continuously running entity are continuously transmitted to other entities through map merging. Therefore, the reconstruction of 3D maps for a single entity can be extended to the reconstruction and fusion of 3D maps for multiple entities. As long as any two entities share a common or previously shared area, the maps of multiple entities can be merged by fusing map feature points, and the relative positions between multiple entities can be determined. This method does not require the introduction of external markers or base stations, reducing deployment and usage costs, and is easily scalable to multiple devices, exhibiting strong environmental adaptability.

[0175] Optionally, for any entity, the entity may also be equipped with an IMU (Inertial Measurement Unit) or other auxiliary sensors to provide additional position information and motion data.

[0176] This method for determining the relative positions of multiple entities can be applied to a system for determining the relative positions of multiple entities, wherein the system includes:

[0177] 1. Visual sensors are used to capture images of the environment and generate image data.

[0178] 2. IMU (optional), used to provide auxiliary positioning information, including acceleration and angular velocity data; it should be noted that the IMU can help improve the positioning accuracy of the system in the case of rapid movement or failure of visual sensors.

[0179] 3. The SLAM algorithm unit processes visual sensor data and IMU data to facilitate real-time 3D map reconstruction and localization. This unit includes front-end and back-end processing modules; the front-end handles feature point extraction, matching, and preliminary pose estimation, while the back-end handles global optimization and map construction.

[0180] 4. Feature point extraction and matching module, used to extract feature points from image data and perform feature point matching.

[0181] 5. Position optimization module, used to optimize and correct the estimated pose and constructed map to ensure the accuracy of relative position.

[0182] 6. A map building module is used to build a sparse 3D point cloud map (referred to as a sparse map in this disclosure) based on the pose and feature points obtained from the SLAM algorithm unit, and to update and maintain the map in real time, and to optimize global consistency by detecting loop closures.

[0183] 7. Map Fusion Module: This module is used to fuse sparse 3D point cloud maps corresponding to multiple entities. Specifically, it calculates coordinate transformation matrices by matching feature points in the sparse 3D point cloud maps of different entities, and transforms the map coordinate system of one entity to the map coordinate system of another entity based on the coordinate transformation matrix, thereby achieving the unification of multi-entity maps.

[0184] 8. The transmission and synchronization module is used to exchange map and location information among multiple entities via wired or wireless communication, ensuring data synchronization among entities. It should be noted that the transmission and synchronization module supports real-time map fusion and location updates, guaranteeing the system's global consistency and real-time performance.

[0185] The method for determining the relative positions of multiple entities disclosed herein has at least the following advantages:

[0186] 1. No external markers required: Eliminates dependence on externally calibrated objects, making it more widely applicable;

[0187] 2. Strong environmental adaptability: It has strong self-adaptability and is suitable for complex and dynamic environments. It is not limited by the installation and maintenance of the calibrated object, nor is it affected by obstruction.

[0188] 3. Rapid integration of new equipment: No complex configuration and calibration process is required when adding new equipment to the corresponding system, which helps to ensure seamless integration and interoperability between new equipment and existing systems.

[0189] The inventors of this disclosure have verified the feasibility of the multi-entity relative position determination method through experiments. Multiple entities equipped with visual sensors were reconstructed in three dimensions and fused with maps in different environments to accurately determine the relative positions between multiple entities. Experimental results show that this method can quickly and accurately determine the relative positions between multiple entities and maintain high real-time performance and stability in dynamic environments.

[0190] To achieve the above embodiments, this disclosure also proposes a multi-entity relative position determination device.

[0191] Figure 6 is a schematic diagram of the structure of a multi-entity relative position determination device provided in another embodiment of this disclosure.

[0192] As shown in Figure 6, the multi-entity relative position determination device 600 includes: an acquisition module 610, an extraction module 620, a construction module 630, and an alignment module 640.

[0193] The acquisition module 610 is used to acquire image data of the environment in which the corresponding entity is located through the visual sensor equipped by each of the multiple entities.

[0194] The extraction module 620 is used to extract feature points from the image data corresponding to any entity, thereby obtaining multiple feature points corresponding to the entity.

[0195] Module 630 is used to construct a sparse map of entities based on multiple feature points corresponding to the entities.

[0196] Alignment module 640 is used to align the sparse map corresponding to any two entities among multiple entities in order to determine the relative position between the two entities.

[0197] Further, in one possible implementation of this disclosure, any two entities include a first entity and a second entity; the alignment module 640 is used to: obtain the similarity between the sparse map of the first entity and the sparse map of the second entity; in response to the similarity being greater than a first preset threshold, align the sparse map of the first entity and the sparse map of the second entity to obtain a first relative pose between the first entity and the second entity; and parse the first relative pose between the first entity and the second entity to obtain the relative position between the first entity and the second entity in any two entities.

[0198] Further, in one possible implementation of this disclosure, the alignment module 640 is configured to: determine a first number of feature points in the sparse map of the first entity that match feature points in the sparse map of the second entity; determine a first coefficient based on the first number and the total number of feature points in the sparse map of the first entity; determine a second number of feature points in the sparse map of the second entity that match feature points in the sparse map of the first entity; determine a second coefficient based on the second number and the total number of feature points in the sparse map of the second entity; and determine the similarity between the sparse maps of the first entity and the sparse map of the second entity based on the first coefficient and / or the second coefficient.

[0199] Furthermore, in one possible implementation of this embodiment, the alignment module 640 is configured to: match each feature point in the sparse map of the first entity and each feature point in the sparse map of the second entity to obtain a matched first feature point pair; determine the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity based on the matched first feature point pair; and determine the first relative pose between the first entity and the second entity based on the transformation relationship.

[0200] Furthermore, in one possible implementation of this disclosure, the apparatus further includes a first determining module and a second determining module. The first determining module is configured to: determine whether a target entity exists among multiple entities in response to a similarity not exceeding a first preset threshold; wherein the similarity between the sparse map of the target entity and the sparse map of the first entity is greater than the first preset threshold, and the similarity between the sparse map of the target entity and the sparse map of the second entity is greater than the first preset threshold; the second determining module is configured to: determine a first relative pose between the first entity and the second entity based on the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the target entity, and the transformation relationship between the entity coordinate system of the second entity and the entity coordinate system of the target entity, in response to the existence of a target entity among multiple entities.

[0201] Furthermore, in one possible implementation of this embodiment, the image data includes multiple consecutive image frames, and each image frame has multiple corresponding feature points; the construction module 630 is used to: sort the multiple consecutive image frames corresponding to the entity according to the order of their acquisition time to obtain an image sorting sequence of the entity; determine the camera coordinate system of the first image frame in the image sorting sequence acquired by the vision sensor equipped with the entity as the entity coordinate system; determine the spatial coordinates of each feature point corresponding to each image frame in the entity coordinate system according to the image sorting sequence of the entity; and construct a sparse map of the entity based on the spatial coordinates of each feature point corresponding to the entity in the corresponding entity coordinate system.

[0202] Further, in one possible implementation of this disclosure embodiment, the construction module 630 is configured to: for any feature point in the first image frame of the image sorting sequence of the entity, determine the spatial coordinates of the feature point in the corresponding entity coordinate system based on the image coordinates of the feature point; for the (i+1)th image frame in the image sorting sequence corresponding to the entity, match multiple feature points of the i-th image frame with multiple feature points of the (i+1)th image frame to obtain a matched second feature point pair; where i is a positive integer not equal to 0; based on the matched second feature point pair, determine the second relative pose of the visual sensor equipped with the entity when acquiring the (i+1)th image frame relative to when acquiring the i-th image frame; for any feature point in the (i+1)th image frame, determine the spatial coordinates of the feature point in the corresponding entity coordinate system based on the second relative pose and the image coordinates of the feature point.

[0203] It should be noted that the foregoing explanation of the multi-entity relative position determination method embodiment also applies to the multi-entity relative position determination device of this embodiment, and will not be repeated here.

[0204] In summary, the multi-entity relative position determination device of this disclosure can automatically determine the relative position between multiple entities by constructing a sparse map for each entity and aligning the sparse maps. It does not rely on external calibration objects, is applicable to various complex and dynamic environments, and has strong adaptability and high flexibility. Since it does not require pre-defining the pattern or features of the calibration objects, it reduces the complexity of algorithm development and adaptation, thereby reducing adaptation costs and time. Even when some feature points are occluded or the environment changes, it can still perform sparse map alignment and positioning using the remaining feature points, improving the robustness, accuracy, and reliability of relative position determination.

[0205] To implement the above embodiments, this disclosure also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the multi-entity relative position determination method provided in the foregoing embodiments.

[0206] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the multi-entity relative position determination method provided in the foregoing embodiments.

[0207] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program that, when executed by a processor, implements the multi-entity relative position determination method provided in the foregoing embodiments.

[0208] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0209] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0210] This disclosure is intended to provide implementation schemes for users to selectively prevent the use or access to their personal information data. Specifically, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0211] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0212] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0213] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0214] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0215] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0216] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0217] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0218] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A method for determining the relative positions of multiple entities, characterized in that, The method includes: Image data of the environment in which the corresponding entity is located is acquired by the visual sensor equipped in each of the multiple entities; For any of the entities, feature points are extracted from the image data corresponding to the entity to obtain multiple feature points corresponding to the entity; Based on multiple feature points corresponding to the entity, a sparse map of the entity is constructed; Align the sparse maps corresponding to any two of the plurality of entities to determine the relative positions between the two entities.

2. The method according to claim 1, characterized in that, The two entities mentioned include the first entity and the second entity; The step of aligning the sparse map corresponding to any two entities among the plurality of entities to determine the relative position between the two entities includes: Obtain the similarity between the sparse map of the first entity and the sparse map of the second entity among any two entities; In response to the similarity being greater than a first set threshold, the sparse maps of the first entity and the second entity are aligned to obtain a first relative pose between the first entity and the second entity. The first relative pose between the first entity and the second entity is analyzed to obtain the relative position between any two entities.

3. The method according to claim 2, characterized in that, The step of obtaining the similarity between the sparse map of the first entity and the sparse map of the second entity among any two entities includes: Determine the first number of feature points in the sparse map of the first entity that match feature points in the sparse map of the second entity; The first coefficient is determined based on the first quantity and the total number of feature points in the sparse map of the first entity; Determine the second number of feature points in the sparse map of the second entity that match feature points in the sparse map of the first entity; The second coefficient is determined based on the second quantity and the total number of feature points in the sparse map of the second entity; Based on the first coefficient and / or the second coefficient, determine the similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities.

4. The method according to claim 2, characterized in that, Aligning the sparse maps of the first entity and the second entity to obtain the first relative pose between the first entity and the second entity includes: The feature points in the sparse map of the first entity and the feature points in the sparse map of the second entity are matched to obtain the first matching feature point pair. Based on the matched first feature point pair, the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity is determined; Based on the transformation relationship, a first relative pose between the first entity and the second entity is determined.

5. The method according to claim 4, characterized in that, The method further includes: In response to the similarity not being greater than the first set threshold, it is determined whether a target entity exists among the plurality of entities; wherein the similarity between the sparse map of the target entity and the sparse map of the first entity is greater than the first set threshold, and the similarity between the sparse map of the target entity and the sparse map of the second entity is greater than the first set threshold. In response to the presence of the target entity among the plurality of entities, a first relative pose between the first entity and the second entity is determined based on the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the target entity, and the transformation relationship between the entity coordinate system of the second entity and the entity coordinate system of the target entity.

6. The method according to any one of claims 1-5, characterized in that, The image data includes multiple consecutive image frames, and correspondingly, each image frame has multiple feature points. The construction of a sparse map corresponding to the entity based on multiple feature points corresponding to the entity includes: The images of the entity are sorted according to the chronological order of their acquisition times to obtain the sorted image sequence of the entity. The camera coordinate system of the first image frame in the image sorting sequence acquired by the visual sensor equipped with the entity is determined as the entity coordinate system; According to the image sorting sequence of the entity, the spatial coordinates of each feature point corresponding to each image frame in the entity's entity coordinate system are determined sequentially. Based on the spatial coordinates of each feature point corresponding to the entity in the corresponding entity coordinate system, a sparse map of the entity is constructed.

7. The method according to claim 6, characterized in that, The step of determining the spatial coordinates of each feature point corresponding to each image frame in the entity's entity coordinate system according to the image sorting sequence of the entity includes: For any feature point in the image frame with sequence number 1 in the image sorting sequence of the entity, the spatial coordinates of the feature point in the corresponding entity coordinate system are determined based on the image coordinates of the feature point. For the (i+1)th image frame in the image sorting sequence corresponding to the entity, multiple feature points of the i-th image frame are matched with multiple feature points of the (i+1)th image frame to obtain a matching second feature point pair; where i is a positive integer not equal to 0. Based on the matched second feature point pair, the second relative pose of the visual sensor equipped with the entity when acquiring the (i+1)th image frame is determined relative to when acquiring the i-th image frame; For any feature point in the (i+1)th image frame, the spatial coordinates of the feature point in the corresponding entity coordinate system are determined based on the second relative pose and the image coordinates of the feature point.

8. A device for determining the relative position of multiple entities, characterized in that, The device includes: The acquisition module is used to acquire image data of the environment in which the corresponding entity is located through the visual sensor equipped by each of the multiple entities; The extraction module is used to extract feature points from the image data corresponding to any one of the entities, thereby obtaining multiple feature points corresponding to the entity. A construction module is used to construct a sparse map of the entity based on multiple feature points corresponding to the entity; An alignment module is used to align the sparse maps corresponding to any two of the plurality of entities to determine the relative positions between the two entities.

9. The apparatus according to claim 8, characterized in that, The two entities include a first entity and a second entity; the alignment module is used for: Obtain the similarity between the sparse map of the first entity and the sparse map of the second entity among any two entities; In response to the similarity being greater than a first set threshold, the sparse maps of the first entity and the second entity are aligned to obtain a first relative pose between the first entity and the second entity. The first relative pose between the first entity and the second entity is analyzed to obtain the relative position between the first entity and the second entity in any two entities.

10. The apparatus according to claim 9, characterized in that, The alignment module is used for: Determine the first number of feature points in the sparse map of the first entity that match feature points in the sparse map of the second entity; The first coefficient is determined based on the first quantity and the total number of feature points in the sparse map of the first entity; Determine the second number of feature points in the sparse map of the second entity that match feature points in the sparse map of the first entity; The second coefficient is determined based on the second quantity and the total number of feature points in the sparse map of the second entity; Based on the first coefficient and / or the second coefficient, determine the similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities.

11. The apparatus according to claim 9, characterized in that, The alignment module is used for: The feature points in the sparse map of the first entity and the feature points in the sparse map of the second entity are matched to obtain the first matching feature point pair. Based on the matched first feature point pair, the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity is determined; Based on the transformation relationship, a first relative pose between the first entity and the second entity is determined.

12. The apparatus according to claim 11, characterized in that, The device further includes: A first determining module is configured to determine whether a target entity exists among the plurality of entities in response to the similarity not being greater than the first preset threshold; wherein the similarity between the sparse map of the target entity and the sparse map of the first entity is greater than the first preset threshold, and the similarity between the sparse map of the target entity and the sparse map of the second entity is greater than the first preset threshold. The second determining module is configured to, in response to the presence of the target entity among the plurality of entities, determine a first relative pose between the first entity and the second entity based on the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the target entity, and the transformation relationship between the entity coordinate system of the second entity and the entity coordinate system of the target entity.

13. The apparatus according to any one of claims 8-12, characterized in that, The image data includes multiple consecutive image frames, and correspondingly, each image frame has multiple feature points. The building module is used for: The images of the entity are sorted according to the chronological order of their acquisition times to obtain the sorted image sequence of the entity. The camera coordinate system of the first image frame in the image sorting sequence acquired by the visual sensor equipped with the entity is determined as the entity coordinate system; According to the image sorting sequence of the entity, the spatial coordinates of each feature point corresponding to each image frame in the entity's entity coordinate system are determined sequentially. Based on the spatial coordinates of each feature point corresponding to the entity in the corresponding entity coordinate system, a sparse map of the entity is constructed.

14. The apparatus according to claim 13, characterized in that, The building module is used for: For any feature point in the image frame with sequence number 1 in the image sorting sequence of the entity, the spatial coordinates of the feature point in the corresponding entity coordinate system are determined based on the image coordinates of the feature point. For the (i+1)th image frame in the image sorting sequence corresponding to the entity, multiple feature points of the i-th image frame are matched with multiple feature points of the (i+1)th image frame to obtain a matching second feature point pair; where i is a positive integer not equal to 0. Based on the matched second feature point pair, the second relative pose of the visual sensor equipped with the entity when acquiring the (i+1)th image frame is determined relative to when acquiring the i-th image frame; For any feature point in the (i+1)th image frame, the spatial coordinates of the feature point in the corresponding entity coordinate system are determined based on the second relative pose and the image coordinates of the feature point.

15. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

17. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images