Array-type multi-vehicle cooperative visual positioning system and method for transporting oversized cargo

Abstract

The present invention relates to the technical field of multi-vehicle cooperation, and discloses an array-type multi-vehicle cooperative visual positioning system and method for transporting oversized cargo, for use in synchronously positioning a plurality of transport vehicles in a vehicle convoy transporting oversized cargo. The system comprises: a positioning module and image markers; a vehicle convoy comprises a plurality of rows of vehicles distributed along a horizontal axis; each row of vehicles comprises two transport vehicles placed side by side; the fronts of all transport vehicles face the same direction; the direction toward which the fronts of the transport vehicles point is defined as forward, and the direction toward which the rears of the transport vehicles point is defined as rearward; the front end of each transport vehicle is provided with a positioning module, and the positioning module comprises a camera, an IMU, and a communication unit. By means of a method in which the two transport vehicles in the frontmost row obtain absolute poses, and each transport vehicle in the subsequent rows observes relative poses, multi-vehicle cooperation is improved, thereby achieving precise real-time positioning of multiple vehicles while enhancing the adaptability of the perception system in degraded environments such as tunnels and poor-signal scenarios.

Description

An array-based multi-vehicle collaborative visual positioning system and method for transporting oversized items Technical Field

[0001] This invention belongs to the field of multi-vehicle collaborative technology, specifically relating to an array-type multi-vehicle collaborative visual positioning system and method for transporting oversized items. Background Technology

[0002] Multi-vehicle cooperative localization is a prerequisite for improving the intelligence level of multi-vehicle cooperative transportation. During operation, due to the large spatial size of each vehicle and the limitations of perception boundaries caused by the array-style transportation layout, it is impossible to directly obtain the relative pose of neighboring vehicles through 3D reconstruction. Furthermore, the localization of each vehicle inevitably contains errors. If the pose relationship between multiple vehicles is obtained directly from the localization information of a single vehicle, the localization error of a single vehicle will be magnified many times over, reducing the estimation accuracy of state quantities such as stress and strain between vehicles and cargo, and increasing the risk of damage to oversized items. On the other hand, when transport vehicles travel in areas with poor signal coverage, such as tunnels, or in degraded environments such as open fields, solutions based on vehicle-mounted RTK or LiDAR may experience degradation in algorithm performance and reliability, or even malfunction. Therefore, existing technologies suffer from insufficient cooperation among multiple vehicles when transporting oversized items. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide an array-type multi-vehicle collaborative visual positioning system and method for transporting oversized items, which solves the problem of insufficient coordination among multiple vehicles when transporting oversized items in existing technologies.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] An array-type multi-vehicle collaborative visual positioning system for transporting oversized items, used to synchronously locate multiple transport vehicles in a convoy transporting oversized items, includes: a positioning module and image tags;

[0006] The convoy consists of multiple rows of vehicles distributed along a horizontal axis. Each row includes two transport vehicles placed side by side. The front of each transport vehicle faces the same direction, with the side with the front of the transport vehicle facing forward being the front and the side with the rear of the transport vehicle facing backward being the rear.

[0007] Each transport vehicle is equipped with a positioning module at the front end, and each positioning module includes a camera, an IMU, and a communication unit;

[0008] The positioning modules on the two transport vehicles in the first row are respectively set as navigation positioning module and absolute positioning module;

[0009] The navigation and positioning module also includes a fusion computing unit;

[0010] The absolute positioning module also includes a positioning calculation unit;

[0011] Image markers are placed at the rear of each of the transport vehicles in front of the last row of transport vehicles;

[0012] The positioning modules on each of the transport vehicles behind the first row of transport vehicles are all set as relative positioning modules. Each relative positioning module also includes a control unit and an illumination module for illuminating the image markers. Each relative positioning module has two cameras, which are used to capture the two image markers on the two transport vehicles in the first row.

[0013] An array-based multi-vehicle cooperative visual positioning method for transporting oversized cargo employs an array-based multi-vehicle cooperative visual positioning system to synchronously locate multiple transport vehicles in a convoy transporting oversized cargo, comprising the following steps:

[0014] Initialization of the array-type multi-vehicle positioning system;

[0015] Absolute pose observation is performed using the navigation positioning module and the absolute positioning module, and the information obtained by the absolute positioning module is transmitted to the navigation positioning module.

[0016] Each relative positioning module performs relative pose observation and transmits the information acquired by each relative positioning module to the navigation positioning module;

[0017] By transmitting various information to the navigation and positioning module and combining it with the spatial constraints of the oversized components, collaborative positioning is achieved.

[0018] Initialization of an array-type multi-vehicle positioning system includes the following steps:

[0019] Establish the coordinate system of each positioning module based on the coordinate system of each IMU;

[0020] RTK was temporarily deployed at the connection points between each transport vehicle and the oversized item, and the vehicle coordinate system of each transport vehicle was established based on each connection point.

[0021] Each transport vehicle drives its corresponding positioning module;

[0022] Obtain the camera coordinates in its own camera coordinate system in each positioning module. Displacement observation points below ,in At the same time, obtain the coordinate system of each IMU in its corresponding positioning module. Displacement observation points below Define the rigid transformation matrix from the coordinates in the positioning module coordinate system to the coordinates in the camera coordinate system as follows: , The expression is as follows:

[0023]

[0024] The displacement observation points of the positioning module in the positioning module coordinate system were obtained respectively. Displacement observation points of RTK in vehicle coordinate system ,in Define the vehicle coordinate system The rigid transformation matrix from the coordinate system of the center coordinate system to the coordinate system of the positioning module is: , The expression is as follows:

[0025]

[0026] The relative positioning module uses two cameras to capture images of two markers on the two vehicles in front of it, and the Perspective-n-Point algorithm is used to obtain the pose of each marker in the camera coordinate system of the camera that captured it. ;

[0027] Assume that the vehicle coordinate system of any transport vehicle with image markers is as follows: Let the vehicle coordinate system of any one of the two transport vehicles in the row after the image marker be . and define median coordinates to The rigid transformation matrix of the coordinate system is Then the coordinate system of the vehicle in which the image marker is set can be calculated. pose in , The expression is as follows:

[0028]

[0029] When oversized items are being installed onto a transport vehicle, in any two adjacent rows of transport vehicles, either the rearmost transport vehicle... Capture any of the front-row transport vehicles Image identification, and obtain image identification on the transport vehicle. The pose of the camera in the camera coordinate system used to capture image markers is: transport vehicle The coordinates in the vehicle coordinate system and the carrier vehicle The rotation transformation matrix between coordinates in the vehicle coordinate system is set as follows: ,in , The expression is as follows:

[0030]

[0031] in, , , and These are all calibration extrinsic parameters of an array-type multi-vehicle positioning system.

[0032] Absolute pose observation is performed using both the navigation and positioning modules, and the information acquired by the absolute positioning module is transmitted to the navigation and positioning module. The specific steps are as follows:

[0033] Based on the cameras in the navigation and absolute positioning modules, the coordinates of environmental landmarks in their respective camera coordinate systems are obtained. ,in ;

[0034] By calibrating extrinsic parameters, the coordinates of environmental landmarks in the vehicle coordinate systems of the navigation and positioning modules and the absolute positioning module are obtained. ,in ;

[0035] Measurements are taken using the IMUs in the navigation positioning module and the absolute positioning module, respectively.

[0036] Obtained from the absolute positioning module and IMU measurements Transmitted to the navigation and positioning module. ,in This represents the acceleration measured by the IMU in the vehicle coordinate system. This represents the angular velocity measured by the IMU in the vehicle coordinate system.

[0037] Each relative positioning module performs relative pose observation and transmits the information acquired by each relative positioning module to the navigation positioning module. The specific steps are as follows:

[0038] The illumination module in the relative positioning module illuminates the image marker in front;

[0039] The corresponding image markers are captured by the cameras in each relative positioning module, and the spatial information of the image markers is obtained. ;

[0040] Spatial information of image identification Transform to the vehicle coordinate system of the carrier vehicle on which the image identifier is installed, and combine with the calibration external parameters. Obtain the rigidity transformation matrix between the vehicle coordinate system of the vehicle on which the current relative positioning module is installed and the vehicle coordinate system of the vehicle on which the image marker is installed. Simultaneously, it obtains IMU measurements in the vehicle coordinate system of the carrier vehicle to which the relative positioning module is installed. ;

[0041] The rigidity change matrix obtained by each relative positioning module and IMU measurements Transmitted to the navigation and positioning module.

[0042] By transmitting various information to the navigation and positioning module and combining it with the spatial constraints of the oversized components, collaborative positioning is performed. The specific steps are as follows:

[0043] Let the number of relative positioning modules be n;

[0044] The state changes of the array-type multi-vehicle positioning system are defined as follows:

[0045]

[0046] in, , ,in ...n;

[0047] When initializing an array-type multi-vehicle positioning system, the vehicle coordinate system of the carrier vehicle on which the navigation positioning module is installed is the global coordinate system. , , Represent The position and pose of the transport vehicle equipped with the relative positioning module and the first row of transport vehicles in the global coordinate system at all times;

[0048] represent The state of the IMU in the vehicle coordinate system of the relative positioning module at any time, including the obtained velocity, angular velocity and its own zero bias;

[0049] IMU pairing is performed using the pre-integration method. Time to The estimation of the vehicle's position and orientation changes over time is as follows:

[0050]

[0051] in, Represents the rotation matrix, translation vector, velocity vector, and zero bias of the gyroscope and accelerometer at time i. This represents the sampling time interval of the IMU, and ~ represents the observation, assuming zero bias. , Satisfying the Wiener process, measuring noise , The discretized zero-mean Gaussian white noise, It represents the change of a certain state variable from time i to time j;

[0052] Construct the IMU measurement residuals based on the IMU measurements:

[0053]

[0054] Each transport vehicle is equipped with a vehicle-to-cargo connection support platform. Oversized items are supported on the vehicle-to-cargo connection support platform. The relationship between the vehicle-to-cargo connection support platform and the oversized items can be simplified to an articulated relationship, thus allowing the measurement of the displacement difference between the front and rear transport vehicles in any two adjacent rows. ,in Based on the dimensions of oversized items, cargo residuals are constructed as follows:

[0055]

[0056] Based on the coordinates of environmental landmarks in the respective vehicle coordinate systems of the navigation and absolute positioning modules. The changes construct the landmark residuals, as shown below:

[0057]

[0058] The relative poses between the transport vehicles are obtained based on the image markers observed by the relative positioning module. The relative pose observation residuals are constructed as follows:

[0059]

[0060] Taking into account all the constructed residuals, the optimal solution is obtained by using the sliding window algorithm and factor graph optimization. This allows for collaborative positioning of multiple vehicles.

[0061] The optimal solution is obtained by using the sliding window algorithm and factor graph optimization. The specific steps are as follows:

[0062] A prior factor is defined as a prior constraint for keyframes in the sliding window algorithm. After each update and optimization, the covariance of the pose of the next keyframe is obtained using a marginalization method and used as a prior factor for the next round of optimization, thus constructing the prior factor marginalization residual. ;

[0063] Solving for the optimal The calculation formula is as follows:

[0064]

[0065] in, Represents the Huber norm. Represents Mahalanobis distance, Represents the residual of IMU measurements. Represents landmark residuals, Represents the residual value of goods. Represents the marginalized residuals of prior factors. This represents the relative pose observation residual.

[0066] The beneficial effects of this invention are:

[0067] This invention provides an array-type multi-vehicle collaborative visual positioning system and method for transporting oversized cargo. It employs a multi-source perception method, such as acquiring the absolute pose of the two frontmost transport vehicles and observing the relative pose of the transport vehicles in the rear rows, to achieve array-type multi-vehicle collaborative positioning for the collaborative transport of oversized cargo. While achieving accurate real-time positioning of multiple vehicles, it improves the adaptability of the array-type multi-vehicle positioning system in degraded environments such as tunnels and scenarios with poor signal, and also enhances the flexibility of sensor arrangement when facing spatial constraints of irregularly shaped oversized cargo. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 is a flowchart of the array-type multi-vehicle positioning method of the present invention;

[0070] Figure 2 is a schematic diagram showing the positions of the navigation positioning module, absolute positioning module and relative positioning module of the present invention.

[0071] Figure 3 is a schematic diagram of the positional relationship between the image identifier and the relative positioning module of the present invention;

[0072] Figure 4 is a schematic diagram of the rigid transformation matrix between the vehicle coordinate systems of any two adjacent vehicles in the present invention.

[0073] Figure 5 is a schematic diagram of the multi-vehicle cooperative positioning principle of the sliding window algorithm of the present invention. Embodiments of the present invention

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] As shown in Figures 1 to 5, an array-type multi-vehicle collaborative visual positioning system for transporting oversized items is used to synchronously locate multiple vehicles in a convoy transporting oversized items, including: a positioning module and image markers;

[0076] The convoy consists of multiple rows of vehicles distributed along a horizontal axis. Each row includes two transport vehicles placed side by side. The front of each transport vehicle faces the same direction, with the side with the front of the transport vehicle facing forward being the front and the side with the rear of the transport vehicle facing backward being the rear.

[0077] Each transport vehicle is equipped with a positioning module at the front end, and each positioning module includes a camera, an IMU, and a communication unit;

[0078] The positioning modules on the two transport vehicles in the first row are respectively set as navigation positioning module and absolute positioning module;

[0079] The navigation and positioning module also includes a fusion computing unit;

[0080] The absolute positioning module also includes a positioning calculation unit;

[0081] Image markers are placed at the rear of each of the transport vehicles in front of the last row of transport vehicles;

[0082] The positioning modules on each of the transport vehicles behind the first row of transport vehicles are all set as relative positioning modules. Each relative positioning module also includes a control unit and an illumination module for illuminating the image markers. Each relative positioning module has two cameras, which are used to capture the two image markers on the two transport vehicles in the first row.

[0083] Preferably, the fused computing unit can be an NVIDIA Jetson Orin NX fused computing unit;

[0084] Preferably, the positioning computing unit can be an NVIDIA Jetson Orin Nano positioning computing unit;

[0085] Preferably, each communication unit supports at least one or more of Bluetooth, Wi-Fi, ZigBee network and LoRa communication, uses TCP protocol to communicate with the other communication units, and ensures reliable data transmission;

[0086] Preferably, the control unit can be a Raspberry Pi 5B control unit;

[0087] Preferably, the cameras in the navigation positioning module and the absolute positioning module can be Intel D435i depth cameras, which can further improve positioning accuracy;

[0088] Preferably, all cameras in the relative positioning module can be monocular cameras, such as 1080P 30fps industrial cameras with 2.02mm focal length, 130° field of view, and distortion-free lenses. This can reduce the overall cost of the system while ensuring effective capture of image markers.

[0089] Preferably, the image markers are made of glass-based material. The image markers include, but are not limited to, feature markers such as ArUco and AprilTag. As shown in Figure 3, each image marker is different on at least two transport vehicles in the same row. The two image markers in the same row have different meanings, such as representing 1 and 2, to avoid the problem of duplicate or incorrect recognition of image markers and to improve the effectiveness and accuracy of image marker pose estimation.

[0090] Preferably, the lighting module can be an infrared LED to reduce the impact of changes in ambient light on image identification, and its sensing range is greater than that of the camera.

[0091] As shown in Figure 1, an array-based multi-vehicle cooperative visual positioning method for transporting oversized items uses an array-based multi-vehicle cooperative visual positioning system to synchronously locate multiple transport vehicles in a convoy transporting oversized items. The method includes the following steps:

[0092] Initialization of the array-type multi-vehicle positioning system;

[0093] Absolute pose observation is performed using the navigation positioning module and the absolute positioning module, and the information obtained by the absolute positioning module is transmitted to the navigation positioning module.

[0094] Each relative positioning module performs relative pose observation and transmits the information acquired by each relative positioning module to the navigation positioning module;

[0095] By transmitting various information to the navigation and positioning module and combining it with the spatial constraints of the oversized components, collaborative positioning is achieved.

[0096] Initialization of an array-type multi-vehicle positioning system includes the following steps:

[0097] Establish the coordinate system of each positioning module based on the coordinate system of each IMU;

[0098] RTK was temporarily deployed at the connection points between each transport vehicle and the oversized item, and the vehicle coordinate system of each transport vehicle was established based on each connection point.

[0099] Each transport vehicle drives its corresponding positioning module;

[0100] Obtain the camera coordinates in its own camera coordinate system in each positioning module. Displacement observation points below ,in At the same time, obtain the coordinate system of each IMU in its corresponding positioning module. Displacement observation points below Define the rigid transformation matrix from the coordinates in the positioning module coordinate system to the coordinates in the camera coordinate system as follows: , The expression is as follows:

[0101]

[0102] Wherein, the rigid transformation matrix T is derived from the rotation matrix and displacement vector For ease of calculation, in subsequent calculations, the rotation matrix R will be transformed into a rotation vector through a logarithmic mapping in the three-dimensional rotation group SO(3), where... ;

[0103]

[0104] in, This represents a mapping from an antisymmetric matrix to a vector.

[0105] The displacement observation points of the positioning module in the positioning module coordinate system were obtained respectively. Displacement observation points of RTK in vehicle coordinate system ,in Define the vehicle coordinate system The rigid transformation matrix from the coordinate system of the center coordinate system to the coordinate system of the positioning module is: , The expression is as follows:

[0106]

[0107] The relative positioning module uses two cameras to capture images of two markers on the two vehicles in front of it, and the Perspective-n-Point algorithm is used to obtain the pose of each marker in the camera coordinate system of the camera that captured it. ;

[0108] Assume that the vehicle coordinate system of any transport vehicle with image markers is as follows: Let the vehicle coordinate system of any one of the two transport vehicles in the row after the image marker be . and define median coordinates to The rigid transformation matrix of the coordinate system is Then the coordinate system of the vehicle in which the image marker is set can be calculated. pose in , The expression is as follows:

[0109]

[0110] As shown in Figure 4, when oversized items are installed onto a transport vehicle, in any two adjacent rows of transport vehicles, the rear transport vehicle... Capture any of the front-row transport vehicles Image identification, and obtain image identification on the transport vehicle. The pose of the camera in the camera coordinate system used to capture image markers is: transport vehicle The coordinates in the vehicle coordinate system and the carrier vehicle The rotation transformation matrix between coordinates in the vehicle coordinate system is set as follows: ,in , The expression is as follows:

[0111]

[0112] in, , , and These are all calibration extrinsic parameters of an array-type multi-vehicle positioning system.

[0113] Absolute pose observation is performed using both the navigation and positioning modules, and the information acquired by the absolute positioning module is transmitted to the navigation and positioning module. The specific steps are as follows:

[0114] Based on the cameras in the navigation and absolute positioning modules, the coordinates of environmental landmarks in their respective camera coordinate systems are obtained. ,in ;

[0115] By calibrating extrinsic parameters, the coordinates of environmental landmarks in the vehicle coordinate systems of the navigation and positioning modules and the absolute positioning module are obtained. ,in ;

[0116] Measurements are taken using the IMUs in the navigation positioning module and the absolute positioning module, respectively.

[0117] Obtained from the absolute positioning module and IMU measurements Transmitted to the navigation and positioning module. ,in This represents the acceleration measured by the IMU in the vehicle coordinate system. This represents the angular velocity measured by the IMU in the vehicle coordinate system.

[0118] Each relative positioning module performs relative pose observation and transmits the information acquired by each relative positioning module to the navigation positioning module. The specific steps are as follows:

[0119] The illumination module in the relative positioning module illuminates the image marker in front;

[0120] The corresponding image markers are captured by the cameras in each relative positioning module, and the spatial information of the image markers is obtained. ;

[0121] Spatial information of image identification Transform to the vehicle coordinate system of the carrier vehicle on which the image identifier is installed, and combine with the calibration external parameters. Obtain the rigidity transformation matrix between the vehicle coordinate system of the vehicle on which the current relative positioning module is installed and the vehicle coordinate system of the vehicle on which the image marker is installed. ;

[0122] The above conversion process is as follows: , And then the solution is obtained ;

[0123] Simultaneously, IMU measurements in the vehicle coordinate system of the transport vehicle where the relative positioning module is installed are obtained. ;

[0124] The rigidity change matrix obtained by each relative positioning module and IMU measurements Transmitted to the navigation and positioning module.

[0125] By transmitting various information to the navigation and positioning module and combining it with the spatial constraints of the oversized components, collaborative positioning is performed. The specific steps are as follows:

[0126] Let the number of relative positioning modules be n;

[0127] The state changes of the array-type multi-vehicle positioning system are defined as follows:

[0128]

[0129] in, , ,in ...n;

[0130] When initializing an array-type multi-vehicle positioning system, the vehicle coordinate system of the carrier vehicle on which the navigation positioning module is installed is the global coordinate system. , , Represent The position and pose of the transport vehicle equipped with the relative positioning module and the first row of transport vehicles in the global coordinate system at all times;

[0131] represent The state of the IMU in the vehicle coordinate system of the relative positioning module at any time, including the obtained velocity, angular velocity and its own zero bias;

[0132] IMU pairing is performed using the pre-integration method. Time to The estimation of the positional change of the transport vehicle at any given time is as shown in equation (6), where, Represents the rotation matrix, translation vector, velocity vector, and zero bias of the gyroscope and accelerometer at time i. This represents the sampling time interval of the IMU, and ~ represents the observation, assuming zero bias. , Satisfying the Wiener process, measuring noise , The discretized zero-mean Gaussian white noise, It represents the change of a certain state variable from time i to time j;

[0133]

[0134] Based on the IMU measurements, the IMU measurement residuals are constructed (referred to as IMU residuals in Figure 5):

[0135]

[0136] in, Let SO(3) be a composite function of the logarithmic mapping and the antisymmetric matrix-to-vector mapping in the three-dimensional rotation group SO(3), and let the rotation matrix be... Transform into a rotation vector;

[0137] As shown in Figure 2, the oversized cargo is considered as a rigid body model. Each transport vehicle is equipped with a vehicle-cargo connection support platform. The oversized cargo is supported on the vehicle-cargo connection support platform. Therefore, the relationship between the vehicle-cargo connection support platform and the oversized cargo can be simplified to a hinged relationship. Thus, the displacement difference between the front and rear transport vehicles in any two adjacent rows can be obtained. ,in Based on the dimensions of oversized items, cargo residuals are constructed as follows:

[0138]

[0139] Based on the coordinates of environmental landmarks in the respective vehicle coordinate systems of the navigation and absolute positioning modules. The changes construct the landmark residuals, as shown below:

[0140]

[0141] The relative poses between the transport vehicles are obtained based on the image markers observed by the relative positioning module. The relative pose observation residuals (referred to as observation residuals in Figure 5) are constructed as follows:

[0142]

[0143] Taking into account all the constructed residuals, the optimal solution is obtained by using the sliding window algorithm and factor graph optimization. This allows for collaborative positioning of multiple vehicles.

[0144] As shown in Figure 5, the optimal solution is obtained using the sliding window algorithm and factor graph optimization. The specific steps are as follows:

[0145] A prior factor is defined as a prior constraint for keyframes in the sliding window algorithm. After each update and optimization, the covariance of the pose of the next keyframe is obtained using a marginalization method and used as a prior factor for the next round of optimization, thus constructing the prior factor marginalization residual. (Refered as a priori residual in Figure 5);

[0146] Solving for the optimal The calculation formula is as follows:

[0147]

[0148] in, Represents the Huber norm. Represents Mahalanobis distance, Represents the residual of IMU measurements. Represents landmark residuals, Represents the residual value of goods. Represents the marginalized residuals of prior factors. Representing the relative pose observation residual, in the sliding window algorithm, 3 keyframe poses are taken for each transport vehicle.

[0149] This invention provides an array-type multi-vehicle collaborative visual positioning system and method for transporting oversized cargo. It employs a multi-source perception method, such as acquiring the absolute pose of the two frontmost transport vehicles and observing the relative pose of the transport vehicles in the rear rows, to achieve array-type multi-vehicle collaborative positioning for the collaborative transport of oversized cargo. While achieving accurate real-time positioning of multiple vehicles, it improves the adaptability of the array-type multi-vehicle positioning system in degraded environments such as tunnels and scenarios with poor signal, and also enhances the flexibility of sensor arrangement when facing spatial constraints of irregularly shaped oversized cargo.

[0150] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0151] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. An array-type multi-vehicle cooperative visual positioning system for transporting oversized items, used for synchronous positioning of multiple transport vehicles in a convoy transporting oversized items, characterized in that, include: Positioning module, image labeling; The convoy consists of multiple rows of vehicles distributed along a horizontal axis. Each row includes two transport vehicles placed side by side. The front of each transport vehicle faces the same direction, with the side with the front of the transport vehicle facing forward being the front and the side with the rear of the transport vehicle facing backward being the rear. Each transport vehicle is equipped with a positioning module at the front end, and each positioning module includes a camera, an IMU, and a communication unit; The positioning modules on the two transport vehicles in the first row are respectively set as navigation positioning module and absolute positioning module; The navigation and positioning module also includes a fusion computing unit, which is an NVIDIA Jetson Orin NX fusion computing unit; The absolute positioning module also includes a positioning computing unit, which is an NVIDIA Jetson Orin Nano positioning computing unit; Image markers are placed at the rear of each of the transport vehicles in front of the last row of transport vehicles; The positioning modules on each of the transport vehicles behind the first row of transport vehicles are all set as relative positioning modules. Each relative positioning module also includes a control unit and an illumination module for illuminating the image markers. Each relative positioning module has two cameras, which are used to capture the two image markers on the two transport vehicles in the first row.

2. [Corrected according to Rule 91, 13.06.2025] An array-type multi-vehicle collaborative visual positioning method for transporting oversized items, which uses the array-type multi-vehicle collaborative visual positioning system for transporting oversized items as described in claim 1 to synchronously locate multiple transport vehicles in a convoy transporting oversized items, characterized in that, The process includes the following steps: initialization of the array-type multi-vehicle positioning system; absolute pose observation through the navigation positioning module and absolute positioning module, and transmission of information acquired by the absolute positioning module to the navigation positioning module; relative pose observation by each relative positioning module, and transmission of information acquired by each relative positioning module to the navigation positioning module; collaborative positioning by combining the information transmitted to the navigation positioning module with the spatial constraints of the oversized components; and collaborative positioning by combining the information transmitted to the navigation positioning module with the spatial constraints of the oversized components. The specific steps are as follows: Let the number of relative positioning modules be n; define the state change variables of the array-type multi-vehicle positioning system. as follows: in, Where i = 1, 2, j = 1, 2... n; when initializing the array-type multi-vehicle positioning system, the vehicle coordinate system of the carrier vehicle on which the navigation positioning module is installed is the global coordinate system {W}. They represent t respectively k The position and pose of the transport vehicle equipped with the relative positioning module and the first row of transport vehicles in the global coordinate system at all times; Represents t k The state of the IMU in the relative positioning module within the vehicle coordinate system where the relative positioning module is installed includes the acquired velocity, angular velocity, and its own zero bias. The IMU estimates the change in vehicle pose from time i to time j using a pre-integration method, as detailed below: Among them, R i ,p i ,v i ,b g,i ,b a,i The rotation matrix, translation vector, velocity vector, and zero bias of the gyroscope and accelerometer at time i are represented, Δt represents the sampling time interval of the IMU, and ~ represents the observations. The zero bias b is assumed to be... a b g Satisfying the Wiener process, measuring noise η ad η gd For the discretized zero-mean Gaussian white noise, Δ(·) ij This represents the change of a certain state variable from time i to time j; based on the IMU measurements, construct the IMU measurement residuals: Each transport vehicle is equipped with a vehicle-to-cargo connection support platform. Oversized items are supported on the vehicle-to-cargo connection support platform. The relationship between the vehicle-to-cargo connection support platform and the oversized items can be simplified to an articulated relationship, thus allowing the measurement of the displacement difference between the front and rear transport vehicles in any two adjacent rows. Where i = 1, 2, j = 1, 2, the cargo residual is constructed based on the size of the oversized item, as shown below: Based on the coordinates of environmental landmarks in the respective vehicle coordinate systems of the navigation and absolute positioning modules. The changes construct the landmark residuals, as shown below: The relative poses between the transport vehicles are obtained based on the image markers observed by the relative positioning module. The relative pose observation residuals are constructed as follows: Taking into account all the constructed residuals, a sliding window algorithm and factor graph optimization are used to solve for the optimal χ, and then collaborative localization of multiple vehicles is performed. The specific steps for solving for the optimal χ using the sliding window algorithm and factor graph optimization are as follows: Define prior factors as prior constraints for keyframes in the sliding window algorithm. After each update optimization, the covariance of the pose of the next keyframe is obtained using the marginalization method, which is used as the prior factor for the next round of optimization. Construct the prior factor marginalization residual r. p The formula for calculating the optimal χ is as follows: Where ρ(·) represents the Huber norm, ||(·)|| ∑ Represents Mahalanobis distance, Represents the residual of IMU measurements. Represents landmark residuals, Represents cargo residuals, r p Represents the marginalized residuals of prior factors. This represents the relative pose observation residual.

3. The array-based multi-vehicle collaborative visual positioning method for transporting oversized items according to claim 2, characterized in that, Initialization of an array-type multi-vehicle positioning system includes the following steps: Based on the coordinate systems of each IMU, establish the coordinate system of each positioning module. RTK was temporarily deployed at the connection points between each transport vehicle and the oversized item, and the vehicle coordinate system of each transport vehicle was established based on each connection point. Each transport vehicle drives its corresponding positioning module; Obtain the displacement observation points of the camera in each positioning module in its own camera coordinate system {C}. Where i = 1, ..., 50, and simultaneously obtain the displacement observation points of each IMU in its corresponding positioning module coordinate system {I}. Define the rigid transformation matrix from the camera coordinate system to the positioning module coordinate system as follows: The expression is as follows: The displacement observation points of the positioning module in the positioning module coordinate system were obtained respectively. Displacement observation points of RTK in vehicle coordinate system Where i = 1,...,50, the rigid transformation matrix from the coordinates in the positioning module coordinate system to the coordinates in the vehicle coordinate system {V} is defined as follows: The expression is as follows: The relative positioning module uses two cameras to capture images of two markers on the two vehicles in front of it, and the Perspective-n-Point algorithm is used to obtain the pose of each marker in the camera coordinate system of the camera that captured it. Suppose the vehicle coordinate system of any transport vehicle with an image marker is {F}, and the vehicle coordinate system of any one of the two transport vehicles in the row behind the image marker is {R}. Define the rigid transformation matrix from coordinates in {R} to coordinates in {F} as follows: The pose of the image marker in the vehicle coordinate system {F} where the image marker is set can then be calculated. The expression is as follows: When oversized components are installed onto the transport vehicle, in any two adjacent rows of transport vehicles, any transport vehicle j in the rear row captures an image marker of any transport vehicle i in the front row, and obtains the pose of the image marker in the camera coordinate system of the camera used by transport vehicle j to capture the image marker. Let the rotation transformation matrix between the coordinates of vehicle j in the vehicle coordinate system and the coordinates of vehicle i in the vehicle coordinate system be denoted as . Where i = 1, 2, j = 1, 2, The expression is as follows: in, and These are all calibration extrinsic parameters of an array-type multi-vehicle positioning system.

4. The array-type multi-vehicle collaborative visual positioning method for transporting oversized items according to claim 3, characterized in that, Absolute pose observation is performed using both the navigation and positioning modules, and the information acquired by the absolute positioning module is transmitted to the navigation and positioning module. The specific steps are as follows: Based on the cameras in the navigation and absolute positioning modules, the coordinates of environmental landmarks in their respective camera coordinate systems are obtained. Where i = 1, 2; By calibrating extrinsic parameters, the coordinates of environmental landmarks in the vehicle coordinate systems of the navigation and positioning modules and the absolute positioning module are obtained. Where i = 1, 2; Measurements are taken using the IMUs in the navigation positioning module and the absolute positioning module, respectively. Obtained from the absolute positioning module and IMU measurements Transmitted to the navigation and positioning module. Where a V ω represents the acceleration measured by the IMU in the vehicle coordinate system. V This represents the angular velocity measured by the IMU in the vehicle coordinate system.

5. The array-based multi-vehicle collaborative visual positioning method for transporting oversized items according to claim 4, characterized in that, Each relative positioning module performs relative pose observation and transmits the information acquired by each relative positioning module to the navigation positioning module. The specific steps are as follows: The illumination module in the relative positioning module illuminates the image marker in front; The corresponding image markers are captured by the cameras in each relative positioning module, and the spatial information of the image markers is obtained. Spatial information of image identification Transform to the vehicle coordinate system of the carrier vehicle on which the image identifier is installed, and combine with the calibration external parameters. Obtain the rigidity transformation matrix between the coordinates in the vehicle coordinate system of the vehicle to which the current relative positioning module is installed and the coordinates in the vehicle coordinate system of the vehicle to which the image marker is installed. Simultaneously, IMU measurements in the vehicle coordinate system of the transport vehicle where the relative positioning module is installed are obtained. The rigidity change matrix obtained by each relative positioning module and IMU measurements Transmitted to the navigation and positioning module.

Images