Vehicle sensor calibration method, vehicle and computer-readable storage medium

Abstract

The embodiments of the present application provide a vehicle sensor calibration method, a vehicle and a computer-readable storage medium. The vehicle sensor calibration method comprises: during motion of a vehicle, acquiring a first image of ground lane lines captured by a fisheye camera; then, on the basis of a projection of the first image in a first-direction view, acquiring Euler angles of the fisheye camera and the vehicle; acquiring a second image of the ground lane lines and a third image of metal poles captured by the fisheye camera, and projecting the second image and the third image in the first-direction view to obtain a first projection image; on the basis of positions of the ground lane lines and the metal poles in the first projection image, determining a vehicle body position; and finally, on the basis of a vehicle body pose and a projection of a fourth image of the metal poles captured by a LiDAR in the first-direction view, acquiring a pose of the LiDAR relative to the vehicle body. Thus, calibration of vehicle sensors can be implemented without relying on targets and / or alignment devices, thereby achieving low implementation costs.

Description

Calibration methods for vehicle sensors, vehicle, and computer-readable storage media Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a method for calibrating vehicle sensors, a vehicle, and a computer-readable storage medium. Background Technology

[0002] Cameras and LiDAR are the main sensors in intelligent driving systems. The extrinsic parameter calibration results of sensors in intelligent driving vehicles directly affect the performance of the perception, planning, decision-making, and localization modules. To ensure the stability and reliability of intelligent driving systems, vehicle sensors need to be calibrated on the production line during vehicle manufacturing. The extrinsic parameters of these sensors refer to their pose relative to the vehicle body.

[0003] Existing production line calibration systems include pre-designed laser and camera calibration targets with pre-defined features, and target fixing devices. When an autonomous vehicle passes through the production line calibration system, the laser camera sensor extracts and identifies the target features to complete accurate and efficient calibration. The entire process involves the design and integration of calibration, the operation of calibration algorithms, and the automation of the calibration process.

[0004] Intelligent driving systems require high accuracy of sensor extrinsic parameters, and vehicle production lines demand high efficiency and stability at every stage. A highly automated and high-precision sensor calibration solution for the production line is crucial to the product competitiveness of intelligent driving vehicles. Summary of the Invention

[0005] This application provides a method for calibrating vehicle sensors, a vehicle, and a computer-readable storage medium to calibrate the extrinsic parameters of vehicle sensors without relying on a alignment device.

[0006] In a first aspect, embodiments of this application provide a vehicle sensor calibration method, comprising: during vehicle movement, acquiring a first image of a ground lane line captured by a fisheye camera; wherein the ground lane line is a lane line on a straight road segment, the ground lane line includes at least two sets of parallel ground lane lines, each set of ground lane lines includes at least two ground lane lines, and the spacing between adjacent lane lines in each set is equal; obtaining the Euler angles of the fisheye camera and the vehicle based on the projection of the first image in a first directional view; acquiring a second image of the ground lane line captured by the fisheye camera and a third image of a metal rod, and projecting the second image and the third image in the first directional view to obtain a first projection image; wherein the metal rod includes two rows of parallel metal rods, the two rows of metal rods are respectively set on both sides of the straight road segment, and the spacing between adjacent metal rods is equal; determining the vehicle position based on the position of the ground lane line and the metal rod in the first projection image; obtaining the pose of the laser radar relative to the vehicle based on the vehicle pose and the projection of a fourth image of the metal rod captured by the laser radar in the first directional view; wherein the vehicle pose includes the Euler angles of the vehicle and the vehicle position.

[0007] In one possible implementation, the fisheye camera includes four fisheye cameras, including a forward-facing fisheye camera; the Euler angles of the fisheye camera include a pitch angle; during vehicle movement, before acquiring the first image of the ground lane lines captured by the fisheye camera, when the vehicle is stationary, a fifth image of the ground lane lines captured by the forward-facing fisheye camera can also be acquired; the fifth image is projected into the first direction view to obtain a second projection image; when the ground lane lines in the second projection image are in a parallel state, the first pitch angle of the forward-facing fisheye camera is obtained.

[0008] In one possible implementation, the Euler angles of the vehicle include the vehicle body pitch angle; obtaining the Euler angles of the fisheye camera and the vehicle based on the projection of the first image in the first direction view can be as follows: based on the projection of the first image in the first direction view, when the ground lane lines are in a parallel state, the pitch angles of the four fisheye cameras are obtained; wherein, the pitch angles of the four fisheye cameras include the second pitch angle of the forward fisheye camera; based on the second pitch angle and the first pitch angle, the vehicle body pitch angle is calculated; based on the vehicle body pitch angle, the pitch angles of the four fisheye cameras are compensated for.

[0009] In addition, during vehicle movement, images of the road lane lines captured by front and rear pinhole cameras can be acquired. Based on the projection of these images into the first-direction view, the pitch angles of the front and rear pinhole cameras are obtained when the road lane lines are parallel. Furthermore, vehicle pitch angle compensation can be performed on the pitch angles of the front and rear pinhole cameras based on these vehicle pitch angles.

[0010] In one possible implementation, the fisheye camera includes four fisheye cameras, and the Euler angles of the fisheye cameras include yaw angles; the Euler angles of the vehicle include vehicle yaw angles; the Euler angles of the fisheye cameras and the vehicle are obtained based on the projection of the first image in the first direction view as follows: based on the projection of the images of the ground lane lines captured by the four fisheye cameras in the same view, when the ground lane lines satisfy the slope constraint, the yaw angles of the four fisheye cameras are obtained; wherein, the same view is the view in the first direction; based on the yaw angles of the four fisheye cameras and the four fisheye cameras... The images of the road lane lines captured by the four fisheye cameras are projected onto the same view, and the pose of the four fisheye cameras relative to the vehicle body is kept unchanged to obtain the first vehicle yaw angle. The first vehicle yaw angle is injected into the four fisheye cameras, and the first vehicle yaw angle is adjusted according to the projection of the images of the road lane lines captured by the four fisheye cameras onto the same view. When the projections of the same road lane line captured by the four fisheye cameras overlap in the same view, the adjusted second vehicle yaw angle is obtained. The vehicle yaw angle compensation is performed on the yaw angle of the four fisheye cameras according to the second vehicle yaw angle.

[0011] In addition, during vehicle movement, images of the ground lane lines captured by the front and rear pinhole cameras can be acquired. Based on the projection of the ground lane lines captured by the front and rear pinhole cameras into the first direction view, when the ground lane lines satisfy the slope constraint, the yaw angle of the front and rear pinhole cameras is obtained. Then, based on the aforementioned second vehicle body yaw angle, vehicle body yaw angle compensation is performed on the yaw angle of the front and rear pinhole cameras.

[0012] In one possible implementation, the fisheye camera includes four fisheye cameras, and the Euler angles of the fisheye camera include the roll angle. The Euler angles of the fisheye camera can be obtained by projecting the first image into the first direction view: based on the projection of the image of the ground lane lines captured by the fisheye camera into the first direction view, when the ground lane lines satisfy the spacing constraint between adjacent lane lines, the roll angle of the fisheye camera can be obtained.

[0013] In addition, based on the projection of the images of the ground lane lines captured by the front and rear pinhole cameras into the first direction view, the roll angle of the front and rear pinhole cameras can be obtained when the ground lane lines meet the spacing constraints between adjacent lane lines.

[0014] In one possible implementation, the vehicle further includes a side-view pinhole camera, and straight line markers are set on the metal rod. After obtaining the Euler angles of the fisheye camera and the vehicle based on the projection of the first image into the first direction view, a sixth image of the straight line markers on the metal rod captured by the side-view pinhole camera can also be obtained. The sixth image is projected onto the plane formed by the metal rod and the straight line markers to obtain a third projection image. When the straight line markers are parallel to the ground in the third projection image and the spacing between the straight line markers meets the spacing constraint, the Euler angles of the side-view pinhole camera are obtained. The Euler angles of the side-view pinhole camera are compensated based on the Euler angles of the vehicle.

[0015] In the aforementioned vehicle sensor calibration method, during vehicle movement, a first image of the ground lane lines captured by a fisheye camera is acquired. Then, based on the projection of the first image into a first-direction view, the Euler angles of the fisheye camera and the vehicle are obtained. Subsequently, a second image of the ground lane lines and a third image of the metal pole captured by the fisheye camera are acquired. The second and third images are projected into the first-direction view to obtain a first projection image. Based on the positions of the ground lane lines and the metal pole in the first projection image, the vehicle position is determined. Finally, based on the vehicle pose and the projection of a fourth image of the metal pole captured by a lidar into the first-direction view, the lidar pose relative to the vehicle is obtained. This method enables sensor calibration of the vehicle without relying on a target and / or alignment device, resulting in lower costs. Furthermore, sensor calibration can be performed during vehicle movement, shortening calibration time and improving production line efficiency. Additionally, the method can locate the vehicle pose, allowing for compensation of the sensor calibration results using the vehicle pose, thus improving sensor calibration accuracy.

[0016] Secondly, embodiments of this application provide a calibration device for a vehicle sensor, comprising: one or more processors; a memory; multiple application programs; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the calibration device, cause the calibration device to perform the method provided in the first aspect.

[0017] Thirdly, embodiments of this application provide a vehicle, including: one or more processors; a memory; multiple application programs; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the vehicle, cause the vehicle to perform the method provided in the first aspect.

[0018] It should be understood that the second and third aspects of the embodiments of this application are consistent with the technical solutions of the first aspect of the embodiments of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be described again.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method provided in the first aspect.

[0020] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when run on a computer, causes the computer to perform the method provided in the first aspect.

[0021] In one possible design, the program in the fifth aspect can be stored wholly or partially on a storage medium packaged with the processor, or it can be stored wholly or partially on a memory not packaged with the processor. Attached Figure Description

[0022] Figure 1 is a structural schematic diagram of a vehicle provided in an embodiment of this application;

[0023] Figure 2 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0024] Figure 3 is a schematic diagram of the layout of a calibration site provided in one embodiment of this application;

[0025] Figure 4 is a schematic diagram of the layout of a metal rod provided in one embodiment of this application;

[0026] Figure 5 is a flowchart of a vehicle sensor calibration method provided in one embodiment of this application;

[0027] Figure 6 is a flowchart of a vehicle sensor calibration method provided in another embodiment of this application;

[0028] Figure 7 is a structural schematic diagram of a vehicle provided in another embodiment of this application;

[0029] Figure 8 is a schematic diagram of the structure of a vehicle sensor calibration device provided in one embodiment of this application. Detailed Implementation

[0030] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0031] For the calibration of extrinsic parameters of vehicle sensors, existing technologies offer the following solutions.

[0032] One approach includes: receiving identification information of a target vehicle located on a straightener; obtaining calibration parameters of the target vehicle based on the identification information; controlling the straightener to apply a straightening force to the target vehicle based on cab width information; adjusting the positions of the camera target and radar target on the calibration bracket based on camera and radar installation position information, so that the camera target is opposite to the camera and the radar target is opposite to the radar; establishing a communication connection with the target vehicle based on vehicle communication interface information; calling the camera calibration program based on the camera identification information and communicating with the target vehicle according to the camera calibration program to calibrate the camera; and calling the radar calibration program based on the radar identification information and communicating with the target vehicle according to the radar calibration program to calibrate the radar.

[0033] However, this solution relies on a leveling device, which has a high equipment cost; and it can only perform static calibration, which takes a long time to calibrate a single vehicle.

[0034] Another approach includes: acquiring an image taken by a camera when the vehicle is in the middle of the lane, the image including clear left and right lane lines; processing the image to obtain the equations of the left and right lane lines in the image, calculating the coordinates of the intersection point P0 of the left and right lane lines and the intersection points P1 and P2 of the left and right lane lines with the bottom edge of the image in the image coordinate system; establishing a world coordinate system to obtain the coordinates of the real space points P0', P1', and P2' corresponding to the points P0, P1, and P2 in the image in the world coordinate system; and obtaining the calibration matrix based on the coordinates of the points P0, P1, and P2 in the image coordinate system and the coordinates of the corresponding real space points P0', P1', and P2' in the world coordinate system.

[0035] However, this scheme requires vehicles to be stationary or moving in the center of the lane, which places excessive demands on the operator.

[0036] Another approach includes: Step S1, placing a checkerboard target to ensure good visibility among at least three sensors; Step S2, extracting the corner points of the checkerboard target and selecting three sensors for combined calibration; Step S3, calculating the cumulative coordinate error of the checkerboard target corner points based on multiple observations; Step S4, determining whether to continue moving the checkerboard target to increase the number of observations; for each additional set of observations, the cumulative coordinate error of the checkerboard target corner points under the current set of observations needs to be calculated according to steps S1 to S3; Step S5, jointly optimizing the cumulative coordinate errors of the checkerboard target corner points at different positions to minimize the total calibration error; simultaneously, since different sensor groups form a loop, the rotation matrix of the loop should be an identity matrix, and the translation matrix should be a zero matrix. This approach can be applied to various scenarios and can guarantee the accuracy of the extrinsic parameters between any two sensors.

[0037] Although this scheme does not use a straightening device, the calculated sensor calibration results are coupled with the vehicle body posture error, resulting in poor sensor calibration accuracy.

[0038] In summary, existing methods for calibrating the extrinsic parameters of vehicle sensors have the following problems:

[0039] 1) Production line calibration solutions rely on alignment and measurement devices and high-precision customized targets, which results in high costs;

[0040] 2) Static production line calibration requires measurement and alignment while the vehicle is stationary, which is time-consuming.

[0041] 3) Before vehicle calibration on the production line, the positioning information relying on the inertial navigation system is unavailable. During dynamic calibration, without vehicle positioning information, the sensor calibration results will be coupled with vehicle pose errors, resulting in poor sensor calibration accuracy.

[0042] Based on the above problems, this application provides a calibration method for vehicle sensors, which can calibrate the extrinsic parameters of vehicle sensors without relying on a straightening device.

[0043] The vehicle sensor calibration method provided in this application can be applied to vehicles. For example, Figure 1 is a schematic diagram of a vehicle structure according to one embodiment of this application. In one embodiment, vehicle 100 is configured in a fully or partially automated driving mode. For instance, vehicle 100 can control itself while in automated driving mode, and the current state of vehicle 100 and its surrounding environment can be determined through human intervention. The possible behaviors of at least one other vehicle in the surrounding environment can be determined, and the confidence level corresponding to the probability of that other vehicle performing the possible behavior can be determined. Based on the determined information, vehicle 100 can be controlled. When vehicle 100 is in automated driving mode, vehicle 100 can be configured to operate without human interaction.

[0044] Vehicle 100 may include various subsystems, such as: a mobility system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112, and a user interface 116. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0045] The propulsion system 102 may include components that provide powered motion to the vehicle 100. In one embodiment, the propulsion system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels / tires 121. The engine 118 may be an internal combustion engine, an electric motor, an air-compressed engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. The engine 118 converts the energy source 119 into mechanical energy.

[0046] Examples of energy sources 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 119 may also provide energy to other systems of vehicle 100.

[0047] The transmission 120 can transmit mechanical power from the engine 118 to the wheels 121. The transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 120 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 121.

[0048] Sensor system 104 may include several sensors for sensing information about the environment surrounding vehicle 100. For example, sensor system 104 may include a positioning system 122 (which may be a Global Positioning System (GPS), BeiDou Navigation Satellite System, or other positioning systems), an inertial measurement unit (IMU) 124, radar 126, a laser rangefinder 128, and a camera 130. Sensor system 104 may also include sensors for the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, fuel gauge, and / or oil temperature gauge). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, and / or speed, etc.). This detection and identification is a key function for the safe operation of the autonomous vehicle 100.

[0049] The positioning system 122 can be used to estimate the geographic location of the vehicle 100. The IMU 124 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In one embodiment, the IMU 124 can be a combination of an accelerometer and a gyroscope.

[0050] Radar 126 can use radio signals to sense objects in the surrounding environment of vehicle 100. In some embodiments, in addition to sensing objects, radar 126 can also be used to sense the speed and / or direction of travel of objects. In embodiments of this application, radar 126 may include lidar.

[0051] The laser rangefinder 128 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0052] Camera 130 can be used to capture multiple images of the surrounding environment of vehicle 100. Camera 130 can be a still camera or a video camera. In this embodiment, camera 130 may include a fisheye camera, front and rear pinhole cameras, and side pinhole cameras. Among them, the fisheye camera may include a four-channel fisheye camera.

[0053] The control system 106 controls the operation of the vehicle 100 and its components. The control system 106 may include various elements, including a steering system 132, a throttle 134, a braking unit 136, a computer vision system 140, a route control system 142, and an obstacle avoidance system 144.

[0054] The steering system 132 is operable to adjust the forward direction of the vehicle 100. For example, in one embodiment, it may be a steering wheel system.

[0055] Throttle 134 is used to control the operating speed of engine 118 and thus the speed of vehicle 100.

[0056] Braking unit 136 is used to control the deceleration of vehicle 100. Braking unit 136 can use friction to slow down wheel 121.

[0057] In other embodiments, braking unit 136 can convert the kinetic energy of wheel 121 into electrical current. Braking unit 136 may also take other forms to slow down the rotational speed of wheel 121 to control the speed of vehicle 100.

[0058] Computer vision system 140 is operable to process and analyze images captured by camera 130 to identify objects and / or features in the environment surrounding vehicle 100. The objects and / or features may include traffic signals, road boundaries, and obstacles. Computer vision system 140 may use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, computer vision system 140 may be used to map the environment, track objects, estimate object velocities, and so on.

[0059] The route control system 142 is used to determine the driving route of the vehicle 100. In some embodiments, the route control system 142 may combine data from the sensor system 104, GPS 122 and one or more predetermined maps to determine the driving route for the vehicle 100.

[0060] The obstacle avoidance system 144 is used to identify, assess and avoid or otherwise traverse potential obstacles in the environment of the vehicle 100.

[0061] Of course, in one instance, the control system 106 may include additional or alternative components besides those shown and described. Alternatively, some of the components shown above may be reduced.

[0062] Vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users via peripheral devices 108. Peripheral devices 108 may include a wireless communication system 146, an on-board computer 148, a microphone 150, and / or a speaker 152.

[0063] In some embodiments, peripheral device 108 provides a means for a user of vehicle 100 to interact with user interface 116. For example, on-board computer 148 may provide information to a user of vehicle 100. User interface 116 may also operate on-board computer 148 to receive user input. On-board computer 148 may be operated via a touchscreen. In other cases, peripheral device 108 may provide a means for vehicle 100 to communicate with other devices located within the vehicle. For example, microphone 150 may receive audio (e.g., voice commands or other audio input) from a user of vehicle 100. Similarly, speaker 152 may output audio to a user of vehicle 100.

[0064] The wireless communication system 146 can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 146 can use 3G cellular communication, such as code division multiple access (CDMA), global system for mobile communications (GSM) / GPRS, or fourth-generation (4G) communication, such as LTE, or fifth-generation (5G) communication. The wireless communication system 146 can communicate using WiFi and a wireless local area network (WLAN). In some embodiments, the wireless communication system 146 can communicate directly with devices using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, are also possible. For example, the wireless communication system 146 may include one or more dedicated short-range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0065] Power source 110 can provide power to various components of vehicle 100. In one embodiment, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs can be configured as a power source to provide power to various components of vehicle 100. In some embodiments, power source 110 and energy source 119 can be implemented together, for example, as in some fully electric vehicles.

[0066] Some or all of the functions of vehicle 100 are controlled by computer system 112. Computer system 112 may include at least one processor 113, which executes instructions 115 stored in a non-transitory computer-readable medium such as data memory 114. Computer system 112 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0067] Processor 113 can be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, the processor can be a dedicated device such as an application-specific integrated circuit (ASIC) or other hardware-based processor. Although Figure 1 functionally illustrates the processor, memory, and other elements of computer 110 in the same block, those skilled in the art will understand that the processor, computer, or memory can actually include multiple processors, computers, or memories not stored in the same physical housing. For example, memory can be a hard disk drive or other storage media located in a housing different from computer 110. Therefore, references to processors or computers will be understood to include references to a collection of processors or computers or memories that may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as steering components and deceleration components, may each have their own processor that performs only calculations related to the component's specific function.

[0068] In the various aspects described herein, the processor may be located remotely from the vehicle and communicate wirelessly with the vehicle. In other aspects, some of the processes described herein are executed on a processor located within the vehicle, while others are executed by a remote processor, including taking the necessary steps to perform a single operation.

[0069] In some embodiments, memory 114 may contain instructions 115 (e.g., program logic) that can be executed by processor 113 to perform various functions of vehicle 100, including those described above. Memory 114 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the mobility system 102, sensor system 104, control system 106, and peripheral devices 108.

[0070] In addition to instruction 115, memory 114 may also store data such as road maps, route information, vehicle position, direction, speed, and other such vehicle data, as well as other information. This information can be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0071] In some embodiments, the processor 113 can also execute the vehicle sensor calibration method provided in this application embodiment to calibrate the extrinsic parameters of the vehicle sensor without relying on the alignment device. Specific calibration methods can be found in the following method embodiments, and will not be repeated here for brevity.

[0072] User interface 116 is used to provide information to or receive information from users of vehicle 100. Optionally, user interface 116 may include one or more input / output devices within a set of peripheral devices 108, such as wireless communication system 146, on-board computer 148, microphone 150, and speaker 152.

[0073] Computer system 112 can control the functions of vehicle 100 based on input received from various subsystems (e.g., driving system 102, sensor system 104, and control system 106) and from user interface 116. For example, computer system 112 can utilize input from control system 106 to control steering system 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144. In some embodiments, computer system 112 is operable to provide control over many aspects of vehicle 100 and its subsystems.

[0074] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, memory 114 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0075] Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 1 should not be construed as a limitation on the embodiments of this application.

[0076] The aforementioned vehicle 100 can be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawnmower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, or handcart, etc., and this application embodiment does not impose any special limitations.

[0077] For ease of understanding, the following embodiments of this application will take a vehicle with the structure shown in FIG1 as an example, and in conjunction with the accompanying drawings and application scenarios, will specifically illustrate the vehicle sensor calibration method provided in the embodiments of this application.

[0078] Figure 2 is a schematic diagram of an application scenario provided by an embodiment of this application. The calibration site shown in Figure 2 can be deployed in the production line lane. The circle represents a metal pole with a certain height, and the white line represents the ground lane line. The calibration site only needs to be selected in a straight section of the production line lane. The metal pole and the ground lane line are both markers of the calibration site. The number and spatial position of the markers can be flexibly changed. The vehicle posture reference coordinate system can be flexibly selected, and there is no need for vehicle posture measurement and alignment device.

[0079] Figure 3 is a schematic diagram of the layout of a calibration site provided in an embodiment of this application. As shown in Figure 3, two rows of parallel metal poles are arranged on both sides of the straight section of the calibration site. The spacing between two adjacent metal poles is equal (D3 = D4), and the two rows of metal poles are symmetrical with respect to the center line of the straight section (as shown in Figure 3).

[0080] Four parallel straight lines of the same color and width are painted on the ground of the straight section of the road, as shown in Figure 3. These four lines are l1, l2, l3 and l4. The distance between l1 and l2 is D1, and the distance between l3 and l4 is D2. D1 and D2 are equal.

[0081] Figure 4 is a schematic diagram of the layout of a metal rod according to an embodiment of this application. As shown in Figure 4, multiple sets of straight markers of equal width and spacing, parallel to the ground and of the same color are arranged on the metal rod. The equal width and spacing of the straight markers means that D5=D6=D7, D8=D9. It should also be noted that in this embodiment, a metal rod is used instead of a rod of other materials so that when calibrating the lidar, the metal rod can meet the detection requirements of the lidar, while rods of other materials cannot meet the detection requirements of the lidar.

[0082] The calibration method for vehicle sensors provided in this application embodiment will be described below based on Figures 3 and 4. In this application embodiment, the vehicle sensors may include a fisheye camera, front and rear pinhole cameras, a lidar, and a side pinhole camera, wherein the fisheye camera may include a four-channel fisheye camera. The vehicle sensors are included in the sensor system 104 of the vehicle 100 shown in Figure 1, wherein the fisheye camera, front and rear pinhole cameras, and side pinhole cameras are included in camera 130, and the lidar is included in radar 126.

[0083] The calibration of vehicle sensors may include the calibration of Euler angles for fisheye cameras, front and rear pinhole cameras and side pinhole cameras, as well as the calibration of extrinsic parameters for lidar. Euler angles are a set of three angles used to describe the orientation of an object in three-dimensional space. These three angles may include yaw angle, pitch angle and roll angle.

[0084] Yaw angle refers to the rotation of an object about its vertical axis (usually the Z-axis), which controls the object's left and right turns.

[0085] Pitch angle refers to the rotation of an object around its horizontal axis (usually the Y-axis), controlling the object's vertical tilt.

[0086] Roll angle refers to the rotation of an object about its longitudinal axis (usually the X-axis), controlling the object's roll along its own axis.

[0087] Figure 5 is a flowchart of a vehicle sensor calibration method provided in an embodiment of this application. The vehicle sensor calibration method can be performed by vehicle 100.

[0088] As shown in Figure 5, the calibration method for the vehicle sensors described above may include:

[0089] Step 501: When the vehicle is stationary, acquire an image of the ground lane lines captured by a forward-facing fisheye camera.

[0090] In practice, the vehicle's stationary duration can be 5 seconds. As shown in Figure 3, the ground lane lines are the lane lines on the straight-ahead road section. The aforementioned ground lane lines include at least two sets of mutually parallel ground lane lines, and each set of ground lane lines includes at least two ground lane lines. The spacing between adjacent lane lines in each set is equal.

[0091] Step 502: Project the image of the ground lane lines captured by the forward-facing fisheye camera into the first direction view to obtain a projection image.

[0092] The first directional view can be a bird's-eye view (BEV), but it can also be a view from other directions. This embodiment does not limit the first directional view.

[0093] Step 503: When the ground lane lines in the above projection map are in a parallel state, obtain the first pitch angle of the forward fisheye camera.

[0094] In addition, when the vehicle is stationary, images of the road lane lines and metal poles captured by a fisheye camera can be acquired. These images are then projected into a first-direction view. Based on the positions of the road lane lines and metal poles in the projected image, the vehicle's pose is initialized. The vehicle pose includes the vehicle's position and its Euler angles. Figure 3 shows the origin, X-axis, and Y-axis of the coordinate system. The initial x-coordinate of the vehicle's position can be determined based on the position of the metal poles, and the y-coordinate can be determined based on the position of the road lane lines. The initial Euler angles of the vehicle can be set to 0°.

[0095] As shown in Figure 3, the aforementioned metal poles consist of two parallel rows of metal poles, which are respectively installed on both sides of the straight road section, with the distance between adjacent metal poles being equal.

[0096] Step 504: During vehicle movement, acquire images of the ground lane lines captured by four fisheye cameras.

[0097] Step 505: Based on the projection of the ground lane lines image captured by the four fisheye cameras into the first direction view, when the ground lane lines are in a parallel state, obtain the pitch angle of the four fisheye cameras.

[0098] In this step, the pitch angle of the four fisheye cameras at time t can be represented by θ. t,i Indicates that i is the number of the four fisheye cameras, i = 1, 2, 3, 4. When i = 1, it represents the pitch angle of the forward-facing fisheye camera. That is to say, the pitch angle of the four fisheye cameras includes the second pitch angle θ of the forward-facing fisheye camera. t,1 .

[0099] Step 506: Calculate the vehicle body pitch angle based on the second pitch angle and the first pitch angle.

[0100] In other words, the vehicle's pitch angle can be calculated based on the pitch angle of the forward fisheye camera when the vehicle is moving and the pitch angle of the forward fisheye camera when the vehicle is stationary.

[0101] Specifically, the vehicle pitch angle can be calculated according to equation (1). Vehicle pitch angle = second pitch angle – first pitch angle. (1)

[0102] Step 507: Based on the above vehicle body pitch angle, perform vehicle body pitch angle compensation on the pitch angle of the four fisheye cameras.

[0103] Specifically, the vehicle body pitch angle can be compensated for by equation (2) for the pitch angle of the four fisheye cameras. t,i,c =θ t,i -θ veh,t (2)

[0104] In equation (2), θt,i,c The pitch angle after compensation for the four-channel fisheye camera; θ t,i The pitch angle of the four-channel fisheye camera before compensation, i.e., the pitch angle of the four-channel fisheye camera obtained in step 505; θ veh,t The vehicle pitch angle is obtained in step 506.

[0105] In addition, during vehicle movement, images of the road lane lines captured by the front and rear pinhole cameras can be acquired. Based on the projection of these images into the first-direction view, the pitch angles of the front and rear pinhole cameras are obtained when the road lane lines are parallel. Furthermore, vehicle pitch angle compensation can be performed on the pitch angles of the front and rear pinhole cameras based on the aforementioned vehicle pitch angles. The compensation method is the same as that for fisheye cameras and will not be elaborated further here.

[0106] Step 508: Project the images of the ground lane lines captured by the four fisheye cameras into the same view.

[0107] In practice, the above-mentioned same view is the view in the first direction. For example, the images of the road lane lines captured by four fisheye cameras can be projected into the same BEV.

[0108] Step 509: Based on the projection of the images of the ground lane lines captured by the four fisheye cameras in the same view, when the ground lane lines satisfy the slope constraint, obtain the yaw angle of the four fisheye cameras.

[0109] In some examples, the slope constraint for the aforementioned ground lane lines can be 0 in the same view. Additionally, it should be noted that the yaw angles of the four fisheye cameras obtained in this step are the yaw angles of the fisheye cameras coupled with the vehicle's yaw angle.

[0110] Step 510: Based on the yaw angles of the four fisheye cameras and the projection of the images of the road lane lines captured by the four fisheye cameras in the same view, keep the pose of the four fisheye cameras relative to the vehicle body unchanged, and obtain the first vehicle yaw angle.

[0111] Step 511: The first vehicle yaw angle is injected into the four fisheye cameras. Based on the projection of the images of the ground lane lines captured by the four fisheye cameras in the same view, the first vehicle yaw angle is adjusted.

[0112] The process of injecting the first vehicle body yaw angle into the four-channel fisheye camera can be achieved by subtracting the first vehicle body yaw angle from the yaw angle of the four-channel fisheye camera obtained in step 509.

[0113] Step 512: When the projections of the same ground lane line captured by the four fisheye cameras coincide in the same view, the adjusted second vehicle yaw angle is obtained.

[0114] Step 513: Based on the second vehicle yaw angle mentioned above, compensate the vehicle yaw angle of the four fisheye cameras.

[0115] Specifically, the yaw angle of the four-way fisheye camera can be compensated according to equation (3). ω t,i,c =ω t,i -ω veh,t (3)

[0116] In equation (3), ω t,i,c Yaw angle after compensation for four-way fisheye cameras; ω t,i The yaw angle of the four-channel fisheye camera before compensation, i.e., the yaw angle of the four-channel fisheye camera obtained in step 509; ω veh,t This is the yaw angle of the second vehicle body.

[0117] In addition, during vehicle movement, images of the ground lane lines captured by the front and rear pinhole cameras can be acquired. Based on the projection of the ground lane lines captured by the front and rear pinhole cameras into the first direction view, when the ground lane lines satisfy the slope constraint, the yaw angles of the front and rear pinhole cameras are obtained. Then, based on the aforementioned second vehicle body yaw angle, vehicle body yaw angle compensation is performed on the yaw angles of the front and rear pinhole cameras. The compensation method is the same as that of the fisheye camera, and will not be described in detail here.

[0118] Step 514: Based on the projection of the image of the ground lane lines captured by the fisheye camera into the first direction view, when the ground lane lines satisfy the spacing constraint between adjacent lane lines, obtain the roll angle of the fisheye camera.

[0119] Referring to Figure 3, the ground lane lines include at least two sets of parallel ground lane lines, and each set of ground lane lines includes at least two ground lane lines. Thus, the spacing constraint between adjacent lane lines can be that the spacing between two adjacent lane lines in each set is equal, i.e., D1 = D2 in Figure 3.

[0120] In addition, based on the projection of the images of the ground lane lines captured by the front and rear pinhole cameras into the first direction view, the roll angle of the front and rear pinhole cameras can be obtained when the ground lane lines meet the spacing constraints between adjacent lane lines.

[0121] This completes the calibration of the Euler angles for the fisheye camera, the front and rear pinhole cameras, and the vehicle's Euler angles. In the description of the above embodiments, the order of obtaining the pitch angle first, then the yaw angle, and finally the roll angle is described. However, this does not constitute a limitation of this embodiment, and this embodiment does not limit the order in which the Euler angles are obtained.

[0122] Step 515: After the Euler angle calibration of the fisheye camera is completed, acquire the images of the ground lane lines and metal poles captured by the fisheye camera, and project the images of the ground lane lines and metal poles captured by the fisheye camera again into the first direction view to obtain the projected image.

[0123] Step 516: Determine the vehicle position based on the positions of the above-mentioned ground lane lines and the above-mentioned metal poles in the projection diagram.

[0124] In this way, after obtaining the Euler angles and vehicle position, the initial vehicle pose can be updated to obtain the updated vehicle pose.

[0125] Step 517: Based on the updated vehicle body pose and the projection of the fourth image of the metal rod captured by the LiDAR into the first directional view, obtain the LiDAR pose relative to the vehicle body.

[0126] The pose of the lidar relative to the vehicle body can include the lidar's position relative to the vehicle body and its Euler angles.

[0127] Step 518: Obtain an image of the straight markings on the metal rod taken by a side-view pinhole camera.

[0128] Step 519: Project the image of the straight line marker onto the plane formed by the metal rod and the straight line marker to obtain a projection image.

[0129] Step 520: When the straight line markers in the above projection diagram are parallel to the ground and the spacing between the straight line markers meets the spacing constraint, the Euler angles of the lateral pinhole camera are obtained.

[0130] As shown in Figure 4, multiple sets of straight markers with equal width and spacing are arranged on the metal rod. Therefore, the spacing constraint can be that the spacing between two adjacent sets of straight markers is equal, and the spacing between two adjacent straight markers in each set is equal, that is, D8=D9, D5=D6=D7 in Figure 4.

[0131] Step 521: Compensate the Euler angles of the side-facing pinhole camera based on the Euler angles of the vehicle.

[0132] In practice, steps 518 to 521 can be executed after step 514.

[0133] Step 522: Save the calibration results.

[0134] The aforementioned vehicle sensor calibration method can achieve vehicle sensor calibration without relying on targets and / or calibrators. It only requires a simple modification to a straight section of road during the production process, resulting in low cost. Furthermore, this embodiment can calibrate vehicle sensors during vehicle movement, improving production line efficiency. Additionally, this embodiment exhibits strong robustness; even if some sensors fail, vehicle position and orientation can still be determined.

[0135] As described above, embodiments of this application provide a calibration method for vehicle sensors that can calibrate vehicle sensors without relying on targets and / or calibrators.

[0136] Figure 6 is a flowchart of a vehicle sensor calibration method provided in another embodiment of this application. The method provided in this embodiment can be executed by vehicle 100.

[0137] As shown in Figure 6, the calibration method for the above-mentioned vehicle sensors may include:

[0138] Step 601: During the movement of vehicle 100, acquire the first image of the ground lane lines captured by the fisheye camera.

[0139] The aforementioned ground lane markings are lane markings on straight-ahead road sections. These ground lane markings include at least two sets of parallel ground lane markings, with each set comprising at least two ground lane markings. The spacing between adjacent lane markings in each set is equal. A schematic diagram of the ground lane markings can be found in Figure 3.

[0140] Step 602: Obtain the Euler angles of the fisheye camera and the vehicle based on the projection of the first image into the first direction view.

[0141] In this step, obtaining the Euler angles of the fisheye camera indicates that the calibration of the fisheye camera is complete.

[0142] Step 603: Acquire a second image of the ground lane lines and a third image of the metal pole taken by the fisheye camera, and project the second and third images into a first directional view to obtain a first projection image.

[0143] The aforementioned metal poles consist of two parallel rows of metal poles, which are respectively installed on both sides of the straight road section, with equal spacing between adjacent metal poles. A schematic diagram of the aforementioned metal poles can be found in Figure 3.

[0144] In this step, after the fisheye camera completes calibration, it takes pictures of the ground lane lines and metal poles. The vehicle 100 acquires a second image of the ground lane lines and a third image of the metal poles taken by the fisheye camera. Then, it projects the second and third images into a first direction view to obtain a first projection image.

[0145] Step 604: Determine the vehicle position based on the positions of the ground lane lines and the metal poles in the first projection diagram.

[0146] Step 605: Based on the vehicle body pose and the projection of the fourth image of the metal rod captured by the lidar into the first directional view, obtain the pose of the lidar relative to the vehicle body.

[0147] The vehicle body pose includes the vehicle's Euler angles and the vehicle body position. The vehicle's Euler angles are the Euler angles of the vehicle obtained in step 602, and the vehicle body position is the vehicle body position determined in step 604.

[0148] In some examples, the fisheye camera includes a four-way fisheye camera, which includes a forward-facing fisheye camera; the Euler angles of the fisheye camera include the pitch angle.

[0149] Thus, before step 601, when vehicle 100 is stationary, vehicle 100 can also acquire a fifth image of the ground lane lines captured by the forward-facing fisheye camera; project the fifth image into the first direction view to obtain a second projection image; when the ground lane lines in the second projection image are in a parallel state, obtain the first pitch angle of the forward-facing fisheye camera. This first pitch angle is the pitch angle when the forward-facing fisheye camera is stationary.

[0150] In these examples, the Euler angles of the aforementioned vehicles include the vehicle pitch angle; step 602 may include:

[0151] Based on the projection of the first image into the first direction view, when the ground lane lines are parallel, the pitch angles of the four fisheye cameras are obtained; wherein, the pitch angles of the four fisheye cameras include the second pitch angle of the forward fisheye camera; the second pitch angle is the pitch angle when the forward fisheye camera is moving. Then, based on the second pitch angle and the first pitch angle, the vehicle body pitch angle is calculated, and then, based on the vehicle body pitch angle, vehicle body pitch angle compensation is performed on the pitch angles of the four fisheye cameras.

[0152] For the front and rear pinhole cameras of vehicle 100, the same method can be used to obtain the pitch angle of the front and rear pinhole cameras. For details, please refer to the relevant description in the embodiment shown in Figure 5, which will not be repeated here.

[0153] In other examples, the fisheye camera includes a four-way fisheye camera, and the Euler angles of the fisheye camera include the yaw angle; the Euler angles of the vehicle include the vehicle body yaw angle; in these examples, step 602 may include:

[0154] Based on the projection of the ground lane lines captured by four fisheye cameras onto the same view, the yaw angle of the four fisheye cameras is obtained when the ground lane lines satisfy the slope constraint; wherein the same view is a view in the first direction. Then, based on the yaw angle of the four fisheye cameras and the projection of the ground lane lines captured by the four fisheye cameras onto the same view, while keeping the pose of the four fisheye cameras relative to the vehicle body unchanged, a first vehicle yaw angle is obtained. The first vehicle yaw angle is injected into the four fisheye cameras, and the first vehicle yaw angle is adjusted according to the projection of the ground lane lines captured by the four fisheye cameras onto the same view; when the projections of the same ground lane line captured by the four fisheye cameras onto the same view coincide, a second vehicle yaw angle is obtained. Finally, based on the second vehicle yaw angle, vehicle yaw angle compensation is performed on the yaw angle of the four fisheye cameras.

[0155] For the front and rear pinhole cameras of vehicle 100, the same method can be used to obtain the yaw angle of the front and rear pinhole cameras. For details, please refer to the relevant description in the embodiment shown in Figure 5, which will not be repeated here.

[0156] In some other examples, the fisheye camera includes a four-way fisheye camera, and the Euler angles of the fisheye camera include the roll angle; step 602 may include:

[0157] Based on the projection of the image of the ground lane lines captured by the fisheye camera into the first direction view, the roll angle of the fisheye camera is obtained when the ground lane lines meet the spacing constraints between adjacent lane lines.

[0158] For the front and rear pinhole cameras of vehicle 100, the same method can be used to obtain the roll angle of the front and rear pinhole cameras. For details, please refer to the relevant description in the embodiment shown in Figure 5, which will not be repeated here.

[0159] In some examples, vehicle 100 also includes a side-mounted pinhole camera, and straight line markers are provided on the metal rod; a schematic diagram of the straight line markers on the metal rod can be shown in Figure 4. In these examples, after step 602, vehicle 100 can also acquire a sixth image of the straight line markers on the metal rod taken by the side-mounted pinhole camera; the sixth image is projected onto the plane formed by the metal rod and the straight line markers to obtain a third projection image. When the straight line markers are parallel to the ground in the third projection image, and the spacing between the straight line markers meets the spacing constraint, the Euler angles of the side-mounted pinhole camera are obtained. Finally, the Euler angles of the side-mounted pinhole camera are compensated according to the Euler angles of the vehicle, thereby completing the calibration of the side-mounted pinhole camera.

[0160] In the aforementioned vehicle sensor calibration method, during the movement of the vehicle 100, a first image of the ground lane lines captured by a fisheye camera is acquired. Then, based on the projection of the first image into a first-direction view, the Euler angles of the fisheye camera and the vehicle are obtained. Subsequently, a second image of the ground lane lines and a third image of the metal pole captured by the fisheye camera are acquired. The second and third images are projected into the first-direction view to obtain a first projection image. Based on the positions of the ground lane lines and the metal pole in the first projection image, the vehicle position is determined. Finally, based on the vehicle pose and the projection of a fourth image of the metal pole captured by the lidar into the first-direction view, the lidar pose relative to the vehicle is obtained. This method enables sensor calibration of the vehicle 100 without relying on a target and / or alignment device, resulting in lower costs. Furthermore, sensor calibration can be performed during the movement of the vehicle 100, shortening calibration time and improving production line efficiency. Additionally, the method can locate the vehicle pose, allowing for compensation of the sensor calibration results using the vehicle pose, thus improving sensor calibration accuracy.

[0161] It is understood that some or all of the steps or operations in the above embodiments are merely examples, and other operations or variations thereof can be performed in the embodiments of this application. Furthermore, the steps may be performed in different orders as presented in the above embodiments, and it is not necessary to perform all the operations in the above embodiments.

[0162] It is understood that, in order to achieve the aforementioned functions, the vehicle includes hardware and / or software modules corresponding to the execution of each function. Based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0163] This embodiment can divide the vehicle into functional modules according to the above method embodiment. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0164] Figure 7 is a structural schematic diagram of a vehicle provided in another embodiment of this application. When each functional module is divided according to its corresponding functions, Figure 7 shows a possible composition schematic diagram of the vehicle 700 involved in the above embodiment. As shown in Figure 7, the vehicle 700 may include: an acquisition unit 701 and a processing unit 702.

[0165] The acquisition unit 701 can be used to support the vehicle 700 in executing steps 501, 504, 515, 518, 601, 603, etc., and / or for other processes of the technical solutions described in the embodiments of this application.

[0166] The processing unit 702 can be used to support the vehicle 700 in executing steps 502 to 503, steps 505 to 515, steps 516 to 517, steps 519 to 522, steps 602 to 605, etc., and / or other processes used in the technical solutions described in the embodiments of this application.

[0167] It should be noted that all relevant content of each step involved in the method embodiments shown in Figures 5 and 6 of this application can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0168] The vehicle 700 provided in this embodiment is used to perform the vehicle sensor calibration method provided in the embodiments shown in Figures 5 and 6 of this application, and thus can achieve the same effect as the above method.

[0169] It should be understood that vehicle 700 can correspond to vehicle 100 shown in FIG1. ​​The function of acquisition unit 701 can be implemented by sensor system 104 in vehicle 100 shown in FIG1; the function of processing unit 702 can be implemented by computer system 112 in vehicle 100 shown in FIG1.

[0170] When using integrated modules, vehicle 700 may include a processing module, a storage module, and a communication module.

[0171] The processing module can be used to control and manage the actions of the vehicle 700, for example, it can support the vehicle 700 in executing the steps performed by the acquisition unit 701 and the processing unit 702. The storage module can be used to support the vehicle 700 in storing program code and data. The communication module can be used to support communication between the vehicle 700 and other devices.

[0172] The processing module can be a processor or controller, which can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. For example, the processor can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, and / or a Wi-Fi chip.

[0173] Figure 8 is a schematic diagram of the structure of a vehicle sensor calibration device provided in an embodiment of this application. The vehicle sensor calibration device includes one or more processors; a memory; multiple application programs; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the calibration device, the calibration device performs the vehicle sensor calibration method provided in the embodiments shown in Figures 5 and 6 of this application.

[0174] As shown in Figure 8, the vehicle sensor calibration device 800 may include a memory 801, a processor 802, a communication interface 803, and a bus 804. The memory 801, processor 802, and communication interface 803 are interconnected via the bus 804.

[0175] The memory 801 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).

[0176] The memory 801 can store a program. When the program stored in the memory 801 is executed by the processor 802, the processor 802 and the communication interface 803 are used to execute the various steps of the vehicle sensor calibration method of the present application embodiment.

[0177] Processor 802 is a circuit with signal processing capabilities. In one implementation, processor 802 can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, processor 802 can implement certain functions through the logical relationships of hardware circuits. These logical relationships of hardware circuits are fixed or reconfigurable. For example, processor 802 is a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement some or all of the functions of the above modules. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. The processor 802 is used to execute relevant programs to implement the functions required by the units in the data processing device of this application embodiment, or to execute the vehicle sensor calibration method of this application method embodiment.

[0178] As can be seen, each module in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor types.

[0179] Furthermore, the modules in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these modules are integrated together as a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or for implementing the functions of the modules of the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0180] The communication interface 803 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the calibration device 800 and other devices or communication networks. For example, data can be acquired through the communication interface 803.

[0181] Bus 804 may include a pathway for transmitting information between various components of calibration device 800 (e.g., memory 801, processor 802, communication interface 803).

[0182] It should be noted that although the calibration device 800 shown in Figure 8 only illustrates the memory, processor, and communication interface, those skilled in the art should understand that in actual implementation, the calibration device 800 also includes other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the calibration device 800 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the calibration device 800 may only include the devices necessary for implementing the embodiments of this application, and not necessarily all the devices shown in Figure 8.

[0183] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to execute the method provided in the embodiments shown in Figures 5 and 6 of this application.

[0184] This application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to execute the method provided in the embodiments shown in Figures 5 and 6 of this application.

[0185] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0186] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0187] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0188] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method of calibrating a vehicle sensor, the method comprising: include: During vehicle movement, a first image of the ground lane lines captured by a fisheye camera is acquired; wherein, the ground lane lines are lane lines on a straight road section, the ground lane lines include at least two sets of parallel ground lane lines, each set of ground lane lines includes at least two ground lane lines, and the spacing between adjacent lane lines in each set is equal. Based on the projection of the first image into the first direction view, the Euler angles of the fisheye camera and the vehicle are obtained; A second image of the road lane lines and a third image of the metal poles captured by the fisheye camera are obtained. The second image and the third image are projected into a first directional view to obtain a first projection image. The metal poles include two parallel rows of metal poles, which are respectively set on both sides of the straight road section, and the spacing between adjacent metal poles is equal. The vehicle position is determined based on the positions of the ground lane lines and the metal poles in the first projection diagram; Based on the vehicle body pose and the projection of the fourth image of the metal rod captured by the lidar into the first directional view, the pose of the lidar relative to the vehicle body is obtained; wherein, the vehicle body pose includes the Euler angles of the vehicle and the vehicle body position.

2. The method of claim 1, wherein, The fisheye camera includes four fisheye cameras, and the four fisheye cameras include a forward-facing fisheye camera; the Euler angles of the fisheye camera include pitch angles. Before acquiring the first image of the ground lane lines captured by the fisheye camera during the vehicle's movement, the process also includes: When the vehicle is stationary, acquire a fifth image of the ground lane lines captured by the forward-facing fisheye camera; The fifth image is projected onto the first view to obtain a second projection image; When the ground lane lines in the second projection map are in a parallel state, the first pitch angle of the forward-facing fisheye camera is obtained.

3. The method of claim 2, wherein, The Euler angles of the vehicle include the vehicle body pitch angle; The step of obtaining the Euler angles of the fisheye camera and the vehicle based on the projection of the first image into the first direction view includes: Based on the projection of the first image into the first direction view, when the ground lane lines are in a parallel state, the pitch angle of the four-way fisheye camera is obtained; wherein, the pitch angle of the four-way fisheye camera includes the second pitch angle of the forward fisheye camera. The vehicle body pitch angle is calculated based on the second pitch angle and the first pitch angle; Based on the vehicle body pitch angle, vehicle body pitch angle compensation is performed on the pitch angle of the four fisheye cameras.

4. The method of claim 1, wherein, The fisheye camera includes four fisheye cameras, and the Euler angles of the fisheye camera include the yaw angle; the Euler angles of the vehicle include the vehicle body yaw angle. The step of obtaining the Euler angles of the fisheye camera and the vehicle based on the projection of the first image into the first direction view includes: Based on the projection of the images of the ground lane lines captured by the four fisheye cameras into the same view, when the ground lane lines satisfy the slope constraint, the yaw angle of the four fisheye cameras is obtained; wherein, the same view is a view in the first direction; Based on the yaw angle of the four fisheye cameras and the projection of the images of the road lane lines captured by the four fisheye cameras in the same view, while keeping the pose of the four fisheye cameras relative to the vehicle body unchanged, the first vehicle body yaw angle is obtained. The first vehicle yaw angle is injected into the four fisheye cameras, and the first vehicle yaw angle is adjusted according to the projection of the images of the ground lane lines captured by the four fisheye cameras in the same view. When the projections of the same ground lane line captured by four fisheye cameras coincide in the same view, the adjusted second vehicle yaw angle is obtained. Based on the second vehicle body yaw angle, the yaw angle of the four fisheye cameras is compensated for.

5. The method of claim 1, wherein, The fisheye camera includes four fisheye cameras, and the Euler angles of the fisheye camera include roll angle; The step of obtaining the Euler angles of the fisheye camera based on the projection of the first image into the first direction view includes: Based on the projection of the image of the ground lane lines captured by the fisheye camera into a first-direction view, the roll angle of the fisheye camera is obtained when the ground lane lines satisfy the spacing constraint between adjacent lane lines.

6. The method of claim 1, wherein, The vehicle also includes a side-mounted pinhole camera, and the metal rod is provided with straight line markers; After obtaining the Euler angles of the fisheye camera and the vehicle based on the projection of the first image into the first direction view, the method further includes: Acquire a sixth image of the straight markings on the metal rod captured by the side-mounted pinhole camera; The sixth image is projected onto the plane formed by the metal rod and the straight marker to obtain a third projection image; When the straight line markers are parallel to the ground in the third projection image and the spacing between the straight line markers meets the spacing constraint, the Euler angles of the lateral pinhole camera are obtained. The Euler angles of the lateral pinhole camera are compensated based on the Euler angles of the vehicle.

7. A calibration device for a vehicle sensor, characterized in that, include: One or more processors; Memory; Multiple applications; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the calibration device, cause the calibration device to perform the method as described in any one of claims 1-6.

8. A vehicle, characterized in that, include: One or more processors; Memory; Multiple applications; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the vehicle, cause the vehicle to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-6.

Images