Image display method and apparatus, vehicle image-assisted display system, and vehicle

Abstract

The present application relates to the technical field of intelligent driving, and discloses an image display method and apparatus, a vehicle image-assisted display system, and a vehicle. The image display method comprises: acquiring an image to be displayed; and adding a pre-calibrated grid to the image to be displayed to obtain a grid image, and displaying the grid image, wherein the pre-calibrated grid comprises a plurality of unit grids, each unit grid corresponds to an actual unit grid size, the actual unit grid size is the size of the unit grid in a vehicle coordinate system, and the vehicle coordinate system is a coordinate system established by taking a specific position or component of a vehicle as an origin. Thus, a user can determine the distance between an object in the image and the vehicle simply from the image, thereby assisting the user in operating the vehicle.

Description

Image display method, device, vehicle image-assisted display system and vehicle

[0001] Cross-references to related applications

[0002] This application claims the benefit of Chinese Patent Application No. 202411962438.8, filed on December 30, 2024, the contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of intelligent driving technology, specifically to an image display method, an image display device, a vehicle image-assisted display system, a vehicle, a machine-readable storage medium, and an electronic device. Background Technology

[0004] Vehicle image-assisted display plays a crucial role in modern automotive technology. During driving, vehicle image-assisted display systems utilize image acquisition devices such as cameras to capture the surrounding environment of the vehicle in real time. Some vehicles are also equipped with 360° panoramic imaging systems, where multiple cameras are distributed around the vehicle to collect image data from different angles. After processing, these images are combined to create a complete bird's-eye view of the vehicle's surroundings, which is then displayed on the screen. However, images can only be used to observe the environment; it is impossible to determine the distance of objects in the image from the vehicle solely from the image itself.

[0005] For example, in medium, large, and super-large excavators, the cab is located at the front of the upper part of the machine, creating a significant blind spot. This typically requires the installation of front and rear high-definition cameras or wide-angle cameras around the perimeter to achieve a 360° surround view. However, regardless of whether it's a high-definition or wide-angle camera, the operator cannot determine the distance of objects to the excavator solely from the image. This is especially true for remotely controlled excavators, where the operator can only observe the environment through images. This undoubtedly reduces the operator's confidence, decreases operational efficiency, and can even lead to collisions.

[0006] Therefore, existing image-assisted displays on vehicles cannot determine the distance between objects and vehicles in an image solely from the image itself. Summary of the Invention

[0007] The purpose of this application is to provide an image display method, an image display device, a vehicle image auxiliary display system, a vehicle, a machine-readable storage medium, and an electronic device to solve the problem in the prior art that the distance between an object and a vehicle in an image cannot be determined solely from the image.

[0008] To achieve the above objectives, the first aspect of this application provides an image display method, comprising:

[0009] Get the image to be displayed;

[0010] The pre-calibrated grid is added to the image to be displayed to obtain a grid image, and the grid image is then displayed.

[0011] The pre-calibrated grid includes multiple unit grids, each unit grid having an actual size. The actual size of the unit grid is the size of the unit grid in the vehicle coordinate system, which is a coordinate system established with a specific position or component of the vehicle as the origin.

[0012] In this embodiment of the application, obtaining the image to be displayed includes:

[0013] Obtain the original image;

[0014] The original image is subjected to distortion correction and / or view transformation to obtain the image to be displayed.

[0015] In this embodiment of the application, the image to be displayed is a corrected image;

[0016] The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes:

[0017] The pre-calibrated grid is added to the image to be displayed to obtain a distortion-free grid image;

[0018] The distortion-free mesh image is converted into a distorted image to obtain a mesh image.

[0019] In this embodiment of the application, adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes:

[0020] Determine whether the image to be displayed and the pre-calibrated grid have different image perspectives;

[0021] If it is determined that the image to be displayed and the pre-calibrated grid have different image perspectives, the view of the image to be displayed or the pre-calibrated grid is transformed to obtain an image to be displayed and a grid with the same image perspective.

[0022] The image to be displayed and the grid with the same image perspective are superimposed to obtain a grid image.

[0023] In this embodiment of the application, there are multiple images to be displayed, and each image to be displayed is acquired from a different location;

[0024] The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes:

[0025] Each image to be displayed is given a pre-calibrated grid to obtain multiple initial grid images, each initial grid image having a top-down view.

[0026] The multiple initial grid images are stitched together to obtain a grid image.

[0027] In this embodiment of the application, the method further includes:

[0028] The grid image is subjected to view transformation and / or distortion processing to obtain a transformed grid image.

[0029] In this embodiment of the application, the image view of the grid image is a top-down view, and the method further includes:

[0030] Real-time acquisition of driving speed and vehicle rotation angle;

[0031] Based on the driving speed and the vehicle body rotation angle, the predicted vehicle path is drawn in the grid image to obtain a top view of the path grid.

[0032] In this embodiment of the application, the mesh calibration process includes:

[0033] Obtain the actual calibration size, which is the size of the first quadrilateral in the vehicle coordinate system. The first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle.

[0034] A calibration image is acquired, and the pixels corresponding to the four points are determined in the calibration image to obtain a second quadrilateral. The acquisition position of the calibration image is the same as that of the image to be displayed. At least one pair of opposite sides of the second quadrilateral is parallel to one side of the calibration image.

[0035] According to the preset segmentation rules, segmentation lines are constructed in the second quadrilateral to obtain the first initial grid;

[0036] Based on the actual calibration dimensions, the actual size of each unit grid in the first initial grid is determined to obtain the grid.

[0037] In this embodiment of the application, the step of constructing dividing lines in the second quadrilateral according to a preset dividing rule to obtain a first initial grid includes:

[0038] According to the preset segmentation rules, the second quadrilateral is evenly divided on each side to obtain multiple first segmentation points;

[0039] Connect the corresponding first dividing points on opposite sides of the second quadrilateral to form dividing lines, thus obtaining the first initial grid.

[0040] In this embodiment of the application, the mesh calibration process includes:

[0041] Obtain the actual calibration size, which is the size of the first quadrilateral in the vehicle coordinate system. The first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle.

[0042] A calibration image is acquired, and the pixels corresponding to the four points are determined in the calibration image to obtain a second quadrilateral. The acquisition position of the calibration image is the same as that of the image to be displayed. At least one pair of opposite sides of the second quadrilateral is parallel to one side of the calibration image.

[0043] Transform the second quadrilateral into a perspective view to obtain the third quadrilateral;

[0044] According to the preset segmentation rules, dividing lines are constructed in the third quadrilateral to obtain the second initial grid;

[0045] Based on the actual calibration dimensions, the actual size of each unit grid in the second initial grid is determined to obtain the grid.

[0046] In this embodiment of the application, the step of constructing dividing lines in the third quadrilateral according to a preset dividing rule to obtain a second initial grid includes:

[0047] According to the preset division rules, the edges of the third quadrilateral are evenly divided to obtain multiple second division points;

[0048] Connect the corresponding second dividing points on opposite sides of the third quadrilateral to form dividing lines, thereby obtaining the second initial grid.

[0049] In this embodiment of the application, after uniformly dividing each side of the third quadrilateral according to a preset division rule to obtain a plurality of second division points, the method further includes:

[0050] By perspective transformation of the plurality of second dividing points into the first quadrilateral, a plurality of third dividing points are obtained;

[0051] Connect the corresponding third dividing points on opposite sides of the first quadrilateral to form dividing lines, thus obtaining the third initial grid;

[0052] Based on the actual calibration dimensions, the actual dimensions of the unit grid in the third initial grid are determined to obtain the grid.

[0053] A second aspect of this application provides an image display device, comprising:

[0054] The acquisition module is used to acquire the image to be displayed.

[0055] The display module is used to add a pre-calibrated grid to the image to be displayed to obtain a grid image and display the grid image. The pre-calibrated grid includes multiple unit grids, each unit grid having an actual size. The actual size of the unit grid is the size of the unit grid in the vehicle coordinate system, which is a coordinate system established with a specific position or component of the vehicle as the origin.

[0056] A third aspect of this application provides a vehicle image-assisted display system, including an image processing device, a display device, and at least one image acquisition device installed on a vehicle. The image acquisition device is used to acquire an image to be displayed and send the image to be displayed to the image processing device. The image processing device is used to add a pre-calibrated grid to the image to be displayed to obtain a grid image and send the grid image to the display device for display. The pre-calibrated grid includes multiple unit grids, each unit grid corresponding to an actual unit grid size. The actual unit grid size is the size of the unit grid in the vehicle coordinate system, which is a coordinate system established with a specific position or component of the vehicle as the origin.

[0057] The fourth aspect of this application provides a vehicle, including a vehicle image auxiliary display system, wherein the vehicle image auxiliary display system displays images using the image display method described above.

[0058] A fifth aspect of this application provides an electronic device, the electronic device comprising:

[0059] At least one processor;

[0060] A memory connected to the at least one processor;

[0061] The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the above-described image display method by executing the instructions stored in the memory.

[0062] A sixth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the image display method described above.

[0063] The above technical solution involves acquiring an image to be displayed; adding a pre-calibrated grid to the image to obtain a grid image; and then displaying the grid image. The pre-calibrated grid includes multiple unit grids, each unit grid corresponding to an actual unit grid size. The actual unit grid size is the size of the unit grid in the vehicle coordinate system, which is established with a specific position or component of the vehicle as its origin. By adding the pre-calibrated grid to the image to be displayed, the displayed image contains a grid. The actual size of each unit grid is the size of each unit grid in the vehicle coordinate system. Therefore, based on the number of unit grids between objects and vehicles in the image, the distance between objects and vehicles in the image can be quickly determined. This allows users to judge the distance between objects and vehicles in the image solely from the image itself, aiding in vehicle operation.

[0064] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0065] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0066] Figure 1 schematically illustrates a flowchart of an image display method according to an embodiment of this application;

[0067] Figure 2 schematically illustrates a coordinate system diagram of an excavator according to an embodiment of this application;

[0068] Figure 3 schematically illustrates a comparison diagram of a normal image and a barrel distortion image according to an embodiment of this application;

[0069] Figure 4 schematically illustrates the original view image according to an embodiment of this application;

[0070] Figure 5 schematically shows a top view with grid lines according to an embodiment of this application;

[0071] Figure 6 schematically illustrates an original view image with grid lines according to an embodiment of this application;

[0072] Figure 7 schematically illustrates a 360° panoramic view image with grid lines according to an embodiment of this application;

[0073] Figure 8 schematically illustrates a diagram showing the predicted straight path of the chassis in a top view of a forward-facing camera according to an embodiment of this application;

[0074] Figure 9 schematically illustrates a diagram showing the predicted arc path of the chassis in a top view of a front-view camera according to an embodiment of this application;

[0075] Figure 10 schematically illustrates a 360° surround view of the predicted straight path of the chassis according to an embodiment of this application;

[0076] Figure 11 schematically illustrates a 360° surround view showing the predicted arc path of the chassis according to an embodiment of this application;

[0077] Figure 12 schematically shows a top view with fan-shaped grid lines according to an embodiment of this application;

[0078] Figure 13 schematically illustrates the excavator image display steps according to an embodiment of this application;

[0079] Figure 14 schematically illustrates the electrical connection diagram of an excavator according to an embodiment of this application;

[0080] Figure 15 schematically illustrates a structural diagram of an image display device according to an embodiment of this application;

[0081] Figure 16 schematically illustrates the internal structure of a computer device according to an embodiment of this application.

[0082] Explanation of reference numerals in the attached drawings: 410 - Acquisition module; 420 - Display module; A01 - Processor; A02 - Network interface; A03 - Internal memory; A04 - Display screen; A05 - Input device; A06 - Non-volatile storage medium; B01 - Operating system; B02 - Computer program. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0084] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0085] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0086] Please refer to Figure 1, which schematically illustrates a flowchart of an image display method according to an embodiment of this application. This embodiment provides an image display method, including the following steps:

[0087] Step 210: Obtain the image to be displayed;

[0088] In this embodiment, the image to be displayed can be acquired by an image acquisition device installed on the vehicle, such as a high-definition camera or a wide-angle camera. The image to be displayed can be the original view image acquired by the image acquisition device, or it can be the original view image after view transformation and / or preprocessing. The preprocessing includes noise reduction and distortion correction. For example, when using a standard camera, distortion is generally rare, while wide-angle cameras capture images with obvious barrel distortion. Therefore, after the wide-angle camera captures the image, barrel distortion correction can be performed to obtain the image to be displayed.

[0089] In some embodiments, obtaining the image to be displayed includes the following steps:

[0090] First, obtain the original image;

[0091] In this embodiment, the original image mentioned above can refer to the image acquired by an image acquisition device. The image acquisition device can be a camera.

[0092] Then, distortion correction and / or view transformation are performed on the original image to obtain the image to be displayed.

[0093] In this embodiment, it is considered that some images captured by cameras may have distortion, such as those captured by fisheye cameras and wide-angle cameras. This embodiment mainly uses a wide-angle camera as an example for explanation. The images captured by a wide-angle camera have obvious barrel distortion. Please refer to Figure 3, which schematically shows a comparison between a normal image and a barrel-distorted image according to an embodiment of this application. Therefore, it is necessary to correct the barrel distortion of the original image captured by the wide-angle camera. Specifically, barrel distortion correction can be performed based on the intrinsic parameters and radial distortion parameters of the wide-angle camera. It should be noted that if the intrinsic parameters and radial distortion parameters of the wide-angle camera are unknown, they can be obtained through camera calibration. Common calibration methods include the Tsai two-step method and Zhang's calibration method. The specific calibration process is prior art and will not be described in detail here. The above-mentioned view transformation refers to converting the image perspective of the original image into a top-down view perspective. Specifically, it can be done by converting the image perspective of the original image into a top-down view perspective through a homography matrix. Since the image information displayed from the top-down view perspective is more intuitive than that displayed from the image perspective perspective, it is easier to view the information in the image. The distortion correction and view transformation described above can be used individually or together, depending on the specific circumstances. When used together, the original image can be distorted first to obtain the corrected image, and then the corrected image can be view transformed to obtain the image to be displayed.

[0094] The following is a detailed explanation of the steps for correcting barrel distortion in wide-angle cameras, including the following steps:

[0095] Step 1: Transform the pixel coordinates (u,v) of the pixel plane to the image plane (x,y), that is:

[0096] Among them, f x ,f y ,c x ,c y This refers to the camera's internal parameters.

[0097] Step 2: Perform radial distortion correction on points in the image plane, specifically:

[0098] in, k1, k2, k3 are radial distortion parameters, (x t ,y t () represents a point on the image plane after radial distortion correction.

[0099] Step 3: Transform the points on the corrected image plane to the pixel plane using intrinsic parameters to obtain pixel coordinates (u). t ,v t Specifically, it is obtained through the following formula:

[0100] It should be noted that if a distortion-free image is to be converted into a barrel distortion image, then step 2 is simply the reverse of barrel distortion correction, i.e., it is obtained through the following formula:

[0101] The view transition is explained in detail below:

[0102] For a pixel in an image captured by a camera with coordinates (u1, v1), the imaging viewpoint can be transformed into a top-down viewpoint using the homography matrix H. In this case, the pixel coordinates become (u2, v2), i.e.:

[0103] Because the above equation is scale invariant, it still holds true when matrix H is multiplied by any scaling factor s. Therefore, we can let h... 33 =1, matrix H has only 8 degrees of freedom, that is:

[0104] Simplifying the above equation, we get:

[0105] Since a set of matching points yields two equations, and solving for eight unknowns requires four sets of matching points, we can obtain the following:

[0106] By finding four sets of pixel coordinates and their corresponding top-view pixel coordinates, and substituting them into the above system of equations, the H matrix can be obtained. Then, the coordinates of all pixels in the original image are projected and transformed into top-view pixel coordinates using the H matrix, thus obtaining the image to be displayed.

[0107] By correcting the distortion of the original image, the impact of distortion on image accuracy can be reduced. Performing a view transformation on the original image can make the transformed image more intuitive and improve image display.

[0108] Step 220: Add the pre-calibrated grid to the image to be displayed to obtain a grid image, and display the grid image; the pre-calibrated grid includes multiple unit grids, each unit grid has an actual size, and the actual size of the unit grid is the size of the unit grid in the vehicle coordinate system, which is a coordinate system established with a specific position or component of the vehicle as the origin.

[0109] In this embodiment, when adding the pre-calibrated mesh to the image to be displayed, if the mesh range is large enough, the pre-calibrated mesh can be directly added to the image. If the mesh range is not large enough, the pre-calibrated mesh can be superimposed on the image to be displayed, and then the mesh can be further extended to expand the mesh range in the image. This extension can be achieved by first extending the edges of the mesh, and then constructing dividing lines in the extended portion according to the unit mesh size. The determination of the mesh range size can be achieved by first setting a range threshold and then comparing the mesh range with the threshold. The pre-calibrated mesh has the actual size of a unit mesh, which can visually display the position of an object relative to the vehicle in the vehicle coordinate system. The mesh calibration can be performed in the image view or in the top-down view. The image to be displayed can be from multiple image perspectives, such as a top-down view or the original viewpoint (i.e., the image viewpoint of the image being displayed). When the image view to be displayed is a top view, adding the pre-calibrated grid to the image to be displayed can achieve the display of the grid in the top view; when the image view to be displayed is the original view, adding the pre-calibrated grid to the image to be displayed can achieve the display of the grid in the original view image.

[0110] The pre-calibrated grid can be from either a top-down view or the original image view. For the grid from the original image view, it can be obtained by direct calibration in the calibration image, which is direct and simple. Alternatively, the segmentation points can be obtained first in the calibration image from the top-down view, and then the segmentation points can be transferred to the calibration image for connection to obtain the grid from the original image view. This calibration method allows for more intuitive calibration of the segmentation points in the calibration image from the top-down view, which helps to obtain a more accurate grid.

[0111] In some embodiments, when calibration is performed using a direct calibration method, the calibration process of the mesh includes the following steps:

[0112] First, obtain the actual calibration size, which is the size of the first quadrilateral in the vehicle coordinate system. The first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle.

[0113] In this embodiment, four vertices can be selected on the ground in front of the camera, ensuring that the line connecting two of these vertices is close to and parallel to the edge of the vehicle body, forming the four vertices of a rectangle. The dimensions of each side of the first quadrilateral are determined based on the positions of the four vertices, thus obtaining the actual calibration dimensions. It should be noted that the larger the rectangle, the more accurate the constructed mesh.

[0114] Then, a calibration image is acquired, and the pixel points corresponding to the four points are determined in the calibration image to obtain a second quadrilateral. The acquisition position of the calibration image is the same as that of the image to be displayed, and at least one pair of opposite sides of the second quadrilateral is parallel to one side of the calibration image.

[0115] In this embodiment, the image viewing angle of the calibration image is the original viewing angle, i.e., the image viewing angle of the image being imaged. The calibration image can be an image after distortion correction following image acquisition by the image acquisition device. Without distortion correction, the calibration image is simply the image acquired by the image acquisition device. The fact that the calibration image and the image to be displayed are acquired at the same location means that the position and angle of the acquisition device are the same when both images are acquired. The calibration image can also be the image to be displayed. The second quadrilateral can be a trapezoid, with its top and bottom sides parallel and parallel to one side of the calibration image. It should be noted that either the top or bottom side of the second quadrilateral can be on one side of the calibration image.

[0116] Then, according to the preset segmentation rules, segmentation lines are constructed in the second quadrilateral to obtain the first initial grid;

[0117] In this embodiment, the preset segmentation rule refers to dividing the grid into multiple unit grids. Based on the number of unit grids to be divided, dividing lines are constructed to obtain the first initial grid.

[0118] In some embodiments, constructing dividing lines in the second quadrilateral according to a preset dividing rule to obtain a first initial grid includes:

[0119] The first step is to divide the second quadrilateral evenly according to the preset division rules to obtain multiple first division points;

[0120] In this embodiment, the number of segmentation points on each edge can be determined according to the preset segmentation rules, thereby obtaining multiple first segmentation points.

[0121] The second step is to connect the corresponding first dividing points on opposite sides of the second quadrilateral to form dividing lines, thereby obtaining the first initial grid.

[0122] In this embodiment, the corresponding first dividing points on opposite edges are connected to form dividing lines. The intersection of the dividing lines forms a unit grid to obtain the first initial grid.

[0123] Finally, based on the actual calibration dimensions, the actual size of each unit grid in the first initial grid is determined to obtain the grid.

[0124] In this embodiment, based on the dimensions of each side in the actual calibration dimensions, the dimensions of the corresponding side of the unit grid can be determined accordingly, thereby obtaining the actual dimensions of the unit grid and calibrating the grid.

[0125] For example, taking an excavator as an example, the original image captured by the camera is shown in Figure 4, and the excavator coordinate system is shown in Figure 2. The excavator coordinate system can be established with the projection of the excavator's rotation center onto the ground as the origin, the X-axis as the excavator's forward movement direction, the Y-axis perpendicular to the X-axis and pointing to the left, and the Z-axis perpendicular to the X-axis and pointing upwards. Find the corresponding points A and D in the excavator coordinate system, measure the distance W meters between AD, and then find points B and C in the excavator coordinate system such that line segment AB extends forward perpendicular to the YZ plane and parallel to the X-axis. The distance AB is the excavator's maximum working radius L. Then find the corresponding pixels for B and C in the image; their pixel coordinates are also known, thus obtaining the actual calibration dimensions and the dimensions of the second quadrilateral. Divide AB and CD into L segments evenly in the original image, and divide BC and AD into W segments evenly. Find the pixel coordinates of (L-1) dividing points on edges AB and CD, and the pixel coordinates of (W-1) dividing points on edges BC and AD; connect the corresponding dividing points on edges AB and CD, and the corresponding dividing points on BC and AD. Each unit grid represents a square with a side length of 1 meter in the excavator coordinate system, forming a grid.

[0126] By acquiring the actual calibration dimensions, a calibration image is obtained, and the pixels corresponding to the four points are determined in the calibration image to obtain a second quadrilateral. According to preset segmentation rules, segmentation lines are constructed in the second quadrilateral to obtain a first initial grid. Based on the actual calibration dimensions, the actual size of each unit grid in the first initial grid is determined, thus enabling rapid and accurate grid calibration in the image view. The number of unit grids can be set as needed using preset segmentation rules to meet different user requirements.

[0127] In some embodiments, the mesh calibration process includes the following steps:

[0128] First, obtain the actual calibration size, which is the size of the first quadrilateral in the vehicle coordinate system. The first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle.

[0129] In this embodiment, four vertices can be selected on the ground in front of the camera to form the four vertices of a rectangle. The dimensions of each side of the first quadrilateral are determined based on the positions of the four vertices, thus obtaining the actual calibration dimensions. It should be noted that the larger the rectangle, the more accurate the constructed mesh.

[0130] Then, a calibration image is acquired, and the pixel points corresponding to the four points are determined in the calibration image to obtain a second quadrilateral. The acquisition position of the calibration image is the same as that of the image to be displayed, and at least one pair of opposite sides of the second quadrilateral is parallel to one side of the calibration image.

[0131] In this embodiment, the image viewing angle of the calibration image is the original viewing angle, i.e., the image viewing angle of the image being imaged. The calibration image can be an image after distortion correction following image acquisition by the image acquisition device. Without distortion correction, the calibration image is simply the image acquired by the image acquisition device. The fact that the calibration image and the image to be displayed are acquired at the same location means that the position and angle of the acquisition device are the same when both images are acquired. The calibration image can also be the image to be displayed. The second quadrilateral can be a trapezoid, with its top and bottom sides parallel and parallel to one side of the calibration image. It should be noted that either the top or bottom side of the second quadrilateral can be on one side of the calibration image.

[0132] Then, the second quadrilateral is transformed into a top view to obtain the third quadrilateral;

[0133] In this embodiment, the perspective transformation described above can be to transform the coordinates of all pixels in the second quadrilateral into the pixel coordinates of the top-view perspective through H matrix projection, thereby obtaining the third quadrilateral.

[0134] Then, according to the preset division rules, dividing lines are constructed in the third quadrilateral to obtain the second initial grid;

[0135] In this embodiment, the obtained third quadrilateral is an image viewed from above, and the preset segmentation rule refers to dividing it into multiple unit grids. Based on the number of unit grids to be divided, dividing lines are constructed in the third quadrilateral to obtain a second initial grid.

[0136] In some embodiments, constructing dividing lines in the third quadrilateral according to a preset dividing rule to obtain a second initial grid includes:

[0137] The first step is to uniformly divide each side of the third quadrilateral according to the preset division rules to obtain multiple second division points;

[0138] In this embodiment, the number of segmentation points on each edge can be determined according to the preset segmentation rules, thereby obtaining multiple second segmentation points.

[0139] The second step is to connect the corresponding second dividing points on opposite sides of the third quadrilateral to form dividing lines, thereby obtaining the second initial grid.

[0140] In this embodiment, the corresponding second dividing points on opposite edges are connected to form dividing lines. The intersection of the dividing lines forms a unit grid to obtain the second initial grid.

[0141] Finally, based on the actual calibration dimensions, the actual size of each unit grid in the second initial grid is determined to obtain the grid.

[0142] In this embodiment, based on the dimensions of each side in the actual calibration dimensions, the dimensions of the corresponding sides of the unit grid can be determined accordingly, thereby obtaining the actual dimensions of the unit grid for calibration. This grid is a grid viewed from above.

[0143] For example, taking an excavator as an example, the original image captured by the camera is shown in Figure 4, and the excavator coordinate system is shown in Figure 2. The image height is h pixels and the width is w pixels. A and D are the lower left and lower right corners of the original image. The coordinates of point A in the pixel coordinate system are (0, h), and the coordinates of point D in the pixel coordinate system are (w, h). The excavator coordinate system can be established with the projection of the excavator's rotation center on the ground as the origin, along the excavator's forward movement direction, with the X-axis being horizontal, the Y-axis being perpendicular to the X-axis and pointing to the left, and the Z-axis being perpendicular to the X-axis and pointing upward. Find the corresponding points A and D in the excavator coordinate system, measure the distance W meters between A and D, and then find points B and C in the excavator coordinate system such that line segment AB extends forward perpendicular to the YZ plane and parallel to the X-axis. The distance of AB is the maximum working radius L of the excavator. Then find the corresponding pixel points B and C in the image, whose pixel coordinates are also known, thus obtaining the actual calibration dimensions and the dimensions of the second quadrilateral. The image size ratio is maintained consistent with the rectangle size ratio in the excavator vehicle coordinate system. The perspective transformation of the original image ABCD is transformed into a new rectangle A′B′C′D′. The coordinates of A′ in the pixel coordinate system are (0, h), B′ in the pixel coordinate system are (0, 0), C′ in the pixel coordinate system are (w, 0), and D′ in the pixel coordinate system are (w, h). Therefore, four pairs of matching points AA′, BB′, CC′, and DD′ can be obtained. The homography matrix H of the perspective transformation can also be obtained through these four pairs of matching points. Dividing the image into L rows and W columns according to each unit grid representing a square with a side length of 1 meter in the vehicle coordinate system, the row spacing is h / L pixels and the column spacing is w / W pixels. It is easy to obtain the pixel coordinates of the (L-1) intersection points of the row dividing line on the sides A′B′ and C′D′, and the pixel coordinates of the (W-1) intersection points of the dividing line on the sides B′C′ and A′D′. Please refer to Figure 5, which schematically shows a top view with grid lines according to an embodiment of this application.

[0144] It should be noted that the grid constructed above is a rectangular grid. In some embodiments, in addition to rectangular grids, sector grids can also be displayed in the top view. Taking an excavator as an example, please refer to Figure 12. Figure 12 schematically shows a top view with sector grid lines according to an embodiment of this application. It should be noted that the center of the circle shown in Figure 12 is the camera. In specific implementations, other locations can also be selected as the center. In the above example, the construction of the sector grid includes the following process: after transforming the perspective of ABCD in the original image into a new rectangle A′B′C′D′, the center coordinates are selected in the top view as... Typically, h / L = w / W, meaning the ratio of the number of pixels in the image's length and width directions to the corresponding distance in the excavator's coordinate system is equal. Then, let the radii of the semicircles be r... i =i*h / L, i = 1, 2, 3, ..., construct a series of concentric circles, and then from the coordinates of the center of each circle... Draw (n-1) rays, with the included angle between two adjacent rays being fixed, and divide the semicircle into n sector grids to obtain a sector grid.

[0145] By obtaining the actual calibration dimensions, a calibration image is obtained, and the pixels corresponding to the four points are determined in the calibration image to obtain a second quadrilateral. The second quadrilateral is then transformed into a top view to obtain a third quadrilateral. According to the preset segmentation rules, dividing lines are constructed in the third quadrilateral to obtain a second initial grid. Based on the actual calibration dimensions, the actual size of the unit grid in the second initial grid is determined, thereby obtaining the grid from the top view.

[0146] In some embodiments, after uniformly dividing the third quadrilateral according to a preset division rule to obtain a plurality of second division points, the method further includes the following step:

[0147] First, the multiple second dividing points are perspective-transformed into the first quadrilateral to obtain multiple third dividing points;

[0148] In this embodiment, the perspective transformation described above can be performed by using the inverse matrix of the homography matrix H to transform the perspective back to the original image, that is, to the first quadrilateral, to obtain multiple third segmentation points.

[0149] Then, connect the corresponding third dividing points on opposite sides of the first quadrilateral to form dividing lines and obtain the third initial grid.

[0150] In this embodiment, connecting the third dividing points corresponding to opposite sides forms a grid in the first quadrilateral, resulting in a third initial grid.

[0151] Finally, based on the actual calibration dimensions, the actual size of each unit grid in the third initial grid is determined to obtain the grid.

[0152] In this embodiment, based on the dimensions of each side in the actual calibration dimensions, the dimensions of the corresponding side of the unit grid can be determined accordingly, thereby obtaining the actual dimensions of the unit grid and calibrating the grid.

[0153] For example, in the above example, the pixel coordinates of (L-1) intersection points of the row dividing line on edges A′B′ and C′D′, and the pixel coordinates of (W-1) intersection points of the dividing line on edges B′C′ and A′D′ are obtained. The intersection points obtained above are in the top view. Then, by using the inverse of the homography matrix H, a perspective transformation can be performed to the original image, and connecting the corresponding intersection points of opposite edges can form a grid in the original image. Please refer to Figure 6, which schematically shows an original view image with grid lines according to an embodiment of this application.

[0154] After determining multiple second dividing points on each side of the third quadrilateral according to preset dividing rules, the multiple second dividing points are perspective-transformed into the first quadrilateral to obtain multiple third dividing points. The corresponding third dividing points on opposite sides of the first quadrilateral are connected to form dividing lines to obtain a third initial grid. Based on the actual calibration size, the actual size of the unit grid in the third initial grid is determined, thereby obtaining the grid under the original viewpoint, which helps to display the grid under the original viewpoint.

[0155] It should be noted that, since a top-down view is more intuitive, a top-down view grid is preferred in practical implementation. The fact that the calibration image and the image to be displayed are acquired from the same location means that the calibration image and the image to be displayed are captured from the same location. For example, if the image to be displayed is captured by a camera directly in front of the vehicle, then correspondingly, the calibration image is also captured by a camera directly in front of the vehicle. This ensures the validity of the grid. Adding the pre-calibrated grid to the image to be displayed can be done by overlaying the pre-calibrated grid image onto the image to be displayed, or by drawing the corresponding grid on the image to be displayed based on the pixel coordinates of the pre-calibrated grid, thereby obtaining a grid image for display.

[0156] In some embodiments, for wide-angle cameras, due to distortion, distortion correction can be performed first after the image is captured, i.e., the image to be displayed is a corrected image; correspondingly, adding a pre-calibrated grid to the image to be displayed to obtain a grid image includes:

[0157] First, the pre-calibrated grid is added to the image to be displayed to obtain a distortion-free grid image;

[0158] Then, the distortion-free mesh image is converted into a distorted image to obtain a mesh image.

[0159] In this embodiment, for a wide-angle camera, the wide-angle view can first be corrected to a distortion-free image through barrel distortion correction, that is, the image to be displayed is the corrected image. Then, after displaying grid lines on the distortion-free image, the distortion-free grid image is converted into a barrel distortion image, that is, the grid image is obtained. This achieves the display of the grid in the original distorted image, thus displaying a wider field of view.

[0160] In practice, images captured by a single high-definition camera have two display modes: one displaying grid lines in the original view image and the other displaying grid lines in the top-down view. Wide-angle cameras, on the other hand, have three display modes: one displaying grid lines in the original barrel distortion image, one displaying grid lines in the original view image, and one displaying grid lines in the top-down view. A comparison of the different display modes is shown in Table 1 below, allowing users to choose the appropriate display method based on their needs.

[0161] Table 1: Comparison of Different Display Modes

[0162] In some embodiments, adding a pre-calibrated grid to the image to be displayed to obtain a grid image includes:

[0163] First, determine whether the image to be displayed and the pre-calibrated grid have different image perspectives;

[0164] In this embodiment, the image to be displayed may have multiple image perspectives, such as the original viewpoint or a top-down viewpoint. Similarly, the image perspective of the pre-calibrated grid may also be multiple perspectives. It can be determined whether the two image perspectives are different.

[0165] Then, if it is determined that the image to be displayed and the pre-calibrated grid have different image perspectives, the image to be displayed or the pre-calibrated grid is transformed to obtain an image to be displayed and a grid with the same image perspective.

[0166] In this embodiment, if the two images have different perspectives, it is necessary to unify their perspectives by performing a view transformation on one of them.

[0167] Finally, the image to be displayed and the grid with the same image perspective are superimposed to obtain a grid image.

[0168] In this embodiment, for example, if the image to be displayed is a top view and the pre-calibrated grid is the original viewpoint, the inverse of the homography matrix H can be used to convert the image to be displayed into the original viewpoint, and then superimposed with the pre-calibrated grid to display the grid in the image of the original viewpoint. Alternatively, the homography matrix H can be used to convert the pre-calibrated grid into a top view, and then superimposed with the image to be displayed to display the grid in the image of the top viewpoint.

[0169] It should be noted that if the image to be displayed and the image of the pre-calibrated grid have the same viewing angle, the pre-calibrated grid is directly added to the image to be displayed to obtain a grid image.

[0170] By determining whether the image to be displayed and the pre-calibrated grid have different image perspectives; if it is determined that the image to be displayed and the pre-calibrated grid have different image perspectives, the view of the image to be displayed or the pre-calibrated grid is transformed to make the image perspectives consistent, and then the image to be displayed and the grid with consistent image perspectives are superimposed to ensure that the grid in the obtained grid image is accurate and reliable.

[0171] In some embodiments, the number of images to be displayed is multiple, and the acquisition positions of each image to be displayed are different; the step of adding a pre-calibrated grid to the images to be displayed to obtain a grid image includes:

[0172] First, a pre-calibrated grid is added to each of the images to be displayed to obtain multiple initial grid images, each of which has a top-down view.

[0173] In this embodiment, considering the presence of multiple cameras in the vehicle, multiple images can be acquired simultaneously, meaning there are multiple images to be displayed, each acquired from a different location. During grid display, a pre-calibrated grid is first added to each image to obtain multiple initial grid images. To facilitate image stitching, the initial grid images are all viewed from a top-down perspective.

[0174] Then, the multiple initial grid images are stitched together to obtain a grid image.

[0175] In this embodiment, the above-mentioned stitching can be a stitching of a 360° panoramic image. Please refer to Figure 7, which schematically shows a 360° panoramic image with grid lines according to an embodiment of this application. The above-mentioned stitching can be achieved using existing image stitching techniques, and will not be described in detail here.

[0176] By using a top-down view with grid lines to stitch together a 360° panoramic image, an auxiliary grid can be formed in the 360° panoramic image, allowing the grid to be displayed in the panoramic image and further helping users judge the distance to objects.

[0177] In the above implementation process, an image to be displayed is acquired; a pre-calibrated grid is added to the image to obtain a grid image, which is then displayed. The pre-calibrated grid includes multiple unit grids, each unit grid corresponding to an actual unit grid size. The actual unit grid size is the size of the unit grid in the vehicle coordinate system, which is a coordinate system established with a specific position or component of the vehicle as the origin. By adding the pre-calibrated grid to the image to be displayed, the displayed image contains a grid. The actual size of each unit grid in the grid is the size of each unit grid in the vehicle coordinate system. Therefore, based on the number of unit grids between objects and vehicles in the image, the distance between objects and vehicles in the image can be quickly determined. This allows users to judge the distance between objects and vehicles in the image solely from the image itself, which helps users operate the vehicle.

[0178] In some embodiments, the method further includes: performing view transformation and / or distortion processing on the grid image to obtain a transformed grid image.

[0179] In this embodiment, since the image view of the grid image can be a top view, a view transformation to the original viewpoint can also be performed. That is, by using the inverse matrix of the homography matrix H, the image view of the initial grid image is transformed into the original viewpoint, thus displaying the grid at the original angle. Furthermore, distortion processing can be applied to the grid image, that is, transforming the undistorted image into a barrel-distorted image, specifically the reverse of barrel distortion correction. The above view transformation and distortion processing can be implemented separately or together, depending on the requirements.

[0180] By performing view transformation and / or distortion processing on the grid image, the grid can be displayed in the original viewpoint and in the original barrel distortion image, thereby realizing multiple display modes to meet various user needs.

[0181] In some embodiments, the image view of the grid image is a top-down view, and the method further includes:

[0182] First, the driving speed and vehicle rotation angle are acquired in real time;

[0183] In this embodiment, the aforementioned travel speed can be obtained by real-time detection of the drive wheel rotation speed. The vehicle body rotation angle refers to the angle of the vehicle body relative to the chassis. Taking an excavator as an example, the vehicle body rotation angle θ (the angle of rotation of the excavator body relative to the tracks) and the speed V of the left and right tracks can be obtained in real time.l V R .

[0184] Then, based on the driving speed and the vehicle body rotation angle, the predicted vehicle path is drawn in the grid image to obtain a top view of the path grid.

[0185] In this embodiment, taking an excavator as an example, the chassis's motion path mainly includes two types: linear motion and circular motion. When the speed V of the left and right tracks... l and V R When the speeds are equal, the chassis movement path is a straight line. When the speeds V of the left and right tracks are equal... l and V R When the values ​​are not equal, the chassis motion path is an arc (circumference), and its radius in the vehicle coordinate system is... v l The left track speed value, v r is the speed value of the right track, and l is the distance between the left and right tracks.

[0186] When the chassis's movement path is a straight line, an initial point can be determined in the grid image. The initial point is the midpoint of the line connecting the center points of the tracks. Then, a straight line can be drawn in the grid image, passing through the initial point, with the angle between the line and the vertical direction being the vehicle body rotation angle. This allows the predicted vehicle path to be drawn in the grid image. Please refer to Figures 8 and 10. Figure 8 schematically shows a diagram of the predicted straight path of the chassis displayed in a top view of the forward-looking camera according to an embodiment of this application. Figure 10 schematically shows a diagram of the predicted straight path of the chassis displayed in a 360° surround view according to an embodiment of this application. It should be noted that in Figure 8, the solid line portion represents the top view display portion of the forward-looking camera, while the dashed line portion is not displayed in the top view of the forward-looking camera.

[0187] When the chassis's movement path is circular, the center coordinates in the image pixel coordinate system can be determined first based on the vehicle's rotation angle, the radius of the arc, the relative position of the midpoint of the line connecting the camera and the track center, and the ratio of the number of pixels in the length and width directions of the image to the corresponding distance in the excavator coordinate system. Taking the vehicle as an excavator as an example, please refer to Figure 9. Figure 9 schematically shows a diagram of the predicted arc path of the chassis displayed in the top view of the front-view camera according to an embodiment of this application. In Figure 9, the solid line part is the display part of the top view of the front-view camera, and the dashed line part is not displayed in the top view of the front-view camera. In the above example, the origin of the image coordinate system is set at the upper left corner of the grid image, downward is the y-axis, and right is the x-axis. Then the coordinates of the center of the circle in the pixel coordinate system (O) x O y This can be represented as:

[0188] Where w is the pixel width of the grid image, h is the pixel height of the grid image, W is the distance corresponding to the pixel width in the excavator coordinate system, L is the distance corresponding to the pixel height in the excavator coordinate system, θ is the vehicle rotation angle, R is the radius of the arc in the vehicle coordinate system, and the forward-facing camera is installed to the right front of the midpoint of the line connecting the centers of the tracks. X c Let Y be the distance between the optical center of the forward-looking camera and the midpoint of the line connecting the center of the track along the Y-axis of the excavator coordinate system. c This refers to the distance between the optical center of the forward-looking camera and the midpoint of the line connecting the center of the track in the X-axis direction of the excavator coordinate system. After determining the coordinates of the center, the arc can be drawn in the grid image based on the coordinates of the center in the pixel coordinate system and the pixel distance of the arc radius to obtain the vehicle's predicted path. Please refer to Figure 11, which schematically shows a diagram of the predicted arc path of the chassis displayed in a 360° surround view according to an embodiment of this application.

[0189] It should be noted that when the grid is a sector grid, the grid display in the 360° surround view can be in the form of concentric circles. Taking the vehicle as an excavator as an example, when the grid is a sector grid, the grid display in the 360° surround view can be in the form of concentric circles with the midpoint of the line connecting the center of the tracks as the center.

[0190] By acquiring the driving speed and vehicle rotation angle in real time, and drawing the vehicle prediction path in the grid image based on the driving speed and vehicle rotation angle, the vehicle prediction path can be added to the grid image to display the vehicle prediction motion path in the image, so that the prediction path can be known from the image alone, which helps the user operate the vehicle.

[0191] In some embodiments, the method further includes: performing view transformation and / or distortion processing on the top view of the path grid to obtain a path grid image.

[0192] In this embodiment, since the top view of the path grid is a top view, a view transformation to the original viewpoint can be performed. This is achieved by using the inverse of the homography matrix H to convert the path grid image view to the original viewpoint, displaying the grid and predicted path at the original angle. Furthermore, distortion processing can be applied to the top view of the path grid, transforming the undistorted image into a barrel-distorted image, specifically the reverse of barrel distortion correction. The view transformation and distortion processing described above can be implemented separately or together, depending on the requirements. Thus, the vehicle's predicted path can first be displayed in a top view, and then projected back to the original viewpoint image using the inverse of the homography matrix H. Additionally, if the chassis moves backward, it can be displayed in the image captured by the rear-view camera, which will not be elaborated further here.

[0193] By performing view transformation and / or distortion processing on the top view of the path grid, it is possible to display grid lines and predicted paths in the original view of the top view of the path grid as well as in the original barrel distortion image, thereby realizing multiple display modes to meet various user needs.

[0194] The following is a specific example. Please refer to Figure 13, which schematically illustrates the excavator image display steps according to an embodiment of this application. Taking an excavator as an example, firstly, the original image and data from each encoder are acquired. Then, the barrel distortion of the image captured by the wide-angle camera is corrected. Next, the original view image is converted into a top view. The grid and predicted chassis trajectory are displayed in the top view. The top view is then stitched into a 360° panoramic view. Finally, the top view is converted back into the original view image, and finally, the original view image is converted back into a wide-angle barrel distortion image.

[0195] It should be understood that although the steps in the flowchart of Figure 1 are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in Figure 1 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0196] This embodiment provides a vehicle image auxiliary display system, including an image processing device, a display device, and at least one image acquisition device installed on the vehicle. The image acquisition device is used to acquire an image to be displayed and send the image to be displayed to the image processing device. The image processing device is used to add a pre-calibrated grid to the image to be displayed to obtain a grid image, and send the grid image to the display device for display. The pre-calibrated grid includes multiple unit grids, each unit grid corresponding to an actual unit grid size. The actual unit grid size is the size of the unit grid in the vehicle coordinate system, which is a coordinate system established with a specific position or component of the vehicle as the origin.

[0197] In this embodiment, the image processing device can be an image processing host, and the image acquisition device includes a camera. Taking an excavator as an example, please refer to Figure 14, which schematically shows the electrical connection diagram of an excavator according to an embodiment of this application. The image processing host receives the vehicle body rotation angle θ (the angle of rotation of the excavator body relative to the tracks) and the speed V of the left and right tracks from the excavator controller. l VR The predicted motion path of the grid and chassis is then displayed on the screen. The excavator controller acquires the vehicle's rotation angle via an encoder at the vehicle's slewing center, and the speeds of the left and right tracks via encoders on the left and right tracks.

[0198] In the above implementation process, the image to be displayed is acquired by the image acquisition device and sent to the image processing device. The image processing device adds a pre-calibrated grid to the image to be displayed to obtain a grid image, which is then sent to the display device for display. The pre-calibrated grid includes multiple unit grids, each unit grid corresponding to an actual unit grid size. The actual unit grid size is the size of the unit grid in the vehicle coordinate system, which is a coordinate system established with a specific position or component of the vehicle as the origin. By adding the pre-calibrated grid to the image to be displayed, the displayed image contains a grid. The actual size of each unit grid in the grid is the size of each unit grid in the vehicle coordinate system. Therefore, the distance between the object and the vehicle in the image can be quickly determined based on the number of unit grids between them. This allows the user to determine the distance between the object and the vehicle in the image solely from the image, which helps the user operate the vehicle.

[0199] This embodiment provides a vehicle, including a vehicle image auxiliary display system, wherein the vehicle image auxiliary display system displays images using the image display method described above.

[0200] In this embodiment, the aforementioned vehicles include passenger cars and engineering vehicles, such as skid steer loaders and wheeled excavators. The image display method described above can display a grid in the image. The actual size of each unit grid is the size of each unit grid in the vehicle coordinate system. Therefore, based on the number of unit grids between objects and vehicles in the image, the distance between objects and vehicles in the image can be quickly determined. This allows users to judge the distance between objects and vehicles in the image solely from the image itself, which is helpful for users to operate the vehicle.

[0201] Please refer to Figure 15, which schematically illustrates a structural diagram of an image display device according to an embodiment of this application. This embodiment provides an image display device, including an acquisition module 410 and a display module 420, wherein:

[0202] The acquisition module 410 is used to acquire the image to be displayed;

[0203] Display module 420 is used to add a pre-calibrated grid to the image to be displayed to obtain a grid image and display the grid image; the pre-calibrated grid includes multiple unit grids, each unit grid has an actual size, the actual size of the unit grid is the size of the unit grid in the vehicle coordinate system, the vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin.

[0204] The image display device includes a processor and a memory. The acquisition module 410 and the display module 420 are stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0205] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and image display is achieved by adjusting kernel parameters.

[0206] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0207] This invention provides a machine-readable storage medium storing a program that, when executed by a processor, implements the image display method.

[0208] This invention provides a processor for running a program, wherein the program executes the image display method during runtime.

[0209] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in Figure 16. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor A01, it implements an image display method. The display screen A04 may be a liquid crystal display (LCD) or an electronic ink display. The input device A05 may be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse, etc.

[0210] Those skilled in the art will understand that the structure shown in Figure 16 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0211] In one embodiment, the image display device provided in this application can be implemented as a computer program, which can run on the computer device shown in FIG16. The memory of the computer device can store various program modules that make up the image display device, such as the acquisition module 410 and the display module 420 shown in FIG15. The computer program composed of the various program modules causes the processor to execute the steps in the image display methods of the various embodiments of this application described in this specification.

[0212] The computer device shown in Figure 16 can perform step 210 through the acquisition module 410 in the image display device shown in Figure 15. The computer device can perform step 220 through the display module 420.

[0213] This application provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the above-described image display method by executing the instructions stored in the memory, wherein the processor executes the instructions to perform the following steps:

[0214] Get the image to be displayed;

[0215] The pre-calibrated grid is added to the image to be displayed to obtain a grid image, and the grid image is then displayed.

[0216] The pre-calibrated grid includes multiple unit grids, each unit grid having an actual size. The actual size of the unit grid is the size of the unit grid in the vehicle coordinate system, which is a coordinate system established with a specific position or component of the vehicle as the origin.

[0217] In one embodiment, obtaining the image to be displayed includes:

[0218] Obtain the original image;

[0219] The original image is subjected to distortion correction and / or view transformation to obtain the image to be displayed.

[0220] In one embodiment, the image to be displayed is a corrected image;

[0221] The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes:

[0222] The pre-calibrated grid is added to the image to be displayed to obtain a distortion-free grid image;

[0223] The distortion-free mesh image is converted into a distorted image to obtain a mesh image.

[0224] In one embodiment, adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes:

[0225] Determine whether the image to be displayed and the pre-calibrated grid have different image perspectives;

[0226] If it is determined that the image to be displayed and the pre-calibrated grid have different image perspectives, the view of the image to be displayed or the pre-calibrated grid is transformed to obtain an image to be displayed and a grid with the same image perspective.

[0227] The image to be displayed and the grid with the same image perspective are superimposed to obtain a grid image.

[0228] In one embodiment, the number of images to be displayed is multiple, and the acquisition positions of each image to be displayed are different;

[0229] The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes:

[0230] Each image to be displayed is given a pre-calibrated grid to obtain multiple initial grid images, each initial grid image having a top-down view.

[0231] The multiple initial grid images are stitched together to obtain a grid image.

[0232] In one embodiment, the method further includes:

[0233] The grid image is subjected to view transformation and / or distortion processing to obtain a transformed grid image.

[0234] In one embodiment, the image view of the grid image is a top-down view, and the method further includes:

[0235] Real-time acquisition of driving speed and vehicle rotation angle;

[0236] Based on the driving speed and the vehicle body rotation angle, the predicted vehicle path is drawn in the grid image to obtain a top view of the path grid.

[0237] In one embodiment, the mesh calibration process includes:

[0238] Obtain the actual calibration size, which is the size of the first quadrilateral in the vehicle coordinate system. The first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle.

[0239] A calibration image is acquired, and the pixels corresponding to the four points are determined in the calibration image to obtain a second quadrilateral. The acquisition position of the calibration image is the same as that of the image to be displayed. At least one pair of opposite sides of the second quadrilateral is parallel to one side of the calibration image.

[0240] According to the preset segmentation rules, segmentation lines are constructed in the second quadrilateral to obtain the first initial grid;

[0241] Based on the actual calibration dimensions, the actual size of each unit grid in the first initial grid is determined to obtain the grid.

[0242] In one embodiment, constructing dividing lines in the second quadrilateral according to a preset dividing rule to obtain a first initial grid includes:

[0243] According to the preset segmentation rules, the second quadrilateral is evenly divided on each side to obtain multiple first segmentation points;

[0244] Connect the corresponding first dividing points on opposite sides of the second quadrilateral to form dividing lines, thus obtaining the first initial grid.

[0245] In one embodiment, the mesh calibration process includes:

[0246] Obtain the actual calibration size, which is the size of the first quadrilateral in the vehicle coordinate system. The first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle.

[0247] A calibration image is acquired, and the pixels corresponding to the four points are determined in the calibration image to obtain a second quadrilateral. The acquisition position of the calibration image is the same as that of the image to be displayed. At least one pair of opposite sides of the second quadrilateral is parallel to one side of the calibration image.

[0248] Transform the second quadrilateral into a perspective view to obtain the third quadrilateral;

[0249] According to the preset segmentation rules, dividing lines are constructed in the third quadrilateral to obtain the second initial grid;

[0250] Based on the actual calibration dimensions, the actual size of each unit grid in the second initial grid is determined to obtain the grid.

[0251] In one embodiment, constructing dividing lines in the third quadrilateral according to a preset dividing rule to obtain a second initial grid includes:

[0252] According to the preset division rules, the edges of the third quadrilateral are evenly divided to obtain multiple second division points;

[0253] Connect the corresponding second dividing points on opposite sides of the third quadrilateral to form dividing lines, thereby obtaining the second initial grid.

[0254] In one embodiment, after uniformly dividing the third quadrilateral according to a preset division rule to obtain a plurality of second division points, the method further includes:

[0255] By perspective transformation of the plurality of second dividing points into the first quadrilateral, a plurality of third dividing points are obtained;

[0256] Connect the corresponding third dividing points on opposite sides of the first quadrilateral to form dividing lines, thus obtaining the third initial grid;

[0257] Based on the actual calibration dimensions, the actual dimensions of the unit grid in the third initial grid are determined to obtain the grid.

[0258] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0259] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0260] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0261] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0262] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0263] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0264] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0265] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0266] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image display method, characterized in that, include: Get the image to be displayed; The pre-calibrated grid is added to the image to be displayed to obtain a grid image, and the grid image is then displayed. The pre-calibrated grid includes multiple unit grids, each unit grid having an actual size. The actual size of the unit grid is the size of the unit grid in the vehicle coordinate system, which is a coordinate system established with a specific position or component of the vehicle as the origin.

2. The method according to claim 1, characterized in that, The process of acquiring the image to be displayed includes: Obtain the original image; The original image is subjected to distortion correction and / or view transformation to obtain the image to be displayed.

3. The method according to claim 1, characterized in that, The image to be displayed is a corrected image; The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes: The pre-calibrated grid is added to the image to be displayed to obtain a distortion-free grid image; The distortion-free mesh image is converted into a distorted image to obtain a mesh image.

4. The method according to claim 1, characterized in that, The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes: Determine whether the image to be displayed and the pre-calibrated grid have different image perspectives; If it is determined that the image to be displayed and the pre-calibrated grid have different image perspectives, the view of the image to be displayed or the pre-calibrated grid is transformed to obtain an image to be displayed and a grid with the same image perspective. The image to be displayed and the grid with the same image perspective are superimposed to obtain a grid image.

5. The method according to claim 1, characterized in that, The number of images to be displayed is multiple, and the acquisition positions of each image to be displayed are different; The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes: Each image to be displayed is given a pre-calibrated grid to obtain multiple initial grid images, each initial grid image having a top-down view. The multiple initial grid images are stitched together to obtain a grid image.

6. The method according to claim 1, characterized in that, The method further includes: The grid image is subjected to view transformation and / or distortion processing to obtain a transformed grid image.

7. The method according to claim 1, characterized in that, The image view of the grid image is a top-down view, and the method further includes: Real-time acquisition of driving speed and vehicle rotation angle; Based on the driving speed and the vehicle body rotation angle, the predicted vehicle path is drawn in the grid image to obtain a top view of the path grid.

8. The method according to claim 1, characterized in that, The calibration process of the grid includes: Obtain the actual calibration size, which is the size of the first quadrilateral in the vehicle coordinate system. The first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle. A calibration image is acquired, and the pixels corresponding to the four points are determined in the calibration image to obtain a second quadrilateral. The acquisition position of the calibration image is the same as that of the image to be displayed. At least one pair of opposite sides of the second quadrilateral is parallel to one side of the calibration image. According to the preset segmentation rules, segmentation lines are constructed in the second quadrilateral to obtain the first initial grid; Based on the actual calibration dimensions, the actual size of each unit grid in the first initial grid is determined to obtain the grid.

9. The method according to claim 8, characterized in that, The step of constructing dividing lines in the second quadrilateral according to preset dividing rules to obtain a first initial grid includes: According to the preset segmentation rules, the second quadrilateral is evenly divided on each side to obtain multiple first segmentation points; Connect the corresponding first dividing points on opposite sides of the second quadrilateral to form dividing lines, thus obtaining the first initial grid.

10. The method according to claim 1, characterized in that, The calibration process of the grid includes: Obtain the actual calibration size, which is the size of the first quadrilateral in the vehicle coordinate system. The first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle. A calibration image is acquired, and the pixels corresponding to the four points are determined in the calibration image to obtain a second quadrilateral. The acquisition position of the calibration image is the same as that of the image to be displayed. At least one pair of opposite sides of the second quadrilateral is parallel to one side of the calibration image. Transform the second quadrilateral into a perspective view to obtain the third quadrilateral; According to the preset segmentation rules, dividing lines are constructed in the third quadrilateral to obtain the second initial grid; Based on the actual calibration dimensions, the actual size of each unit grid in the second initial grid is determined to obtain the grid.

11. The method according to claim 10, characterized in that, The step of constructing dividing lines in the third quadrilateral according to preset dividing rules to obtain a second initial grid includes: According to the preset division rules, the edges of the third quadrilateral are evenly divided to obtain multiple second division points; Connect the corresponding second dividing points on opposite sides of the third quadrilateral to form dividing lines, thereby obtaining the second initial grid.

12. The method according to claim 11, characterized in that, After uniformly dividing the third quadrilateral according to a preset division rule to obtain multiple second division points, the method further includes: By perspective transformation of the plurality of second dividing points into the first quadrilateral, a plurality of third dividing points are obtained; Connect the corresponding third dividing points on opposite sides of the first quadrilateral to form dividing lines, thus obtaining the third initial grid; Based on the actual calibration dimensions, the actual dimensions of the unit grid in the third initial grid are determined to obtain the grid.

13. An image display device, characterized in that, include: The acquisition module is used to acquire the image to be displayed. The display module is used to add a pre-calibrated grid to the image to be displayed to obtain a grid image and display the grid image. The pre-calibrated grid includes multiple unit grids, each unit grid having an actual size. The actual size of the unit grid is the size of the unit grid in the vehicle coordinate system, which is a coordinate system established with a specific position or component of the vehicle as the origin.

14. A vehicle image auxiliary display system, characterized in that, The system includes an image processing device, a display device, and at least one image acquisition device mounted on a vehicle. The image acquisition device acquires an image to be displayed and sends the image to the image processing device. The image processing device adds a pre-calibrated grid to the image to be displayed to obtain a grid image, and sends the grid image to the display device for display. The pre-calibrated grid includes multiple unit grids, each unit grid corresponding to an actual unit grid size. The actual unit grid size is the size of the unit grid in the vehicle coordinate system, which is a coordinate system established with a specific position or component of the vehicle as the origin.

15. A vehicle, characterized in that, The system includes a vehicle image auxiliary display system, wherein the vehicle image auxiliary display system displays images using the image display method described in any one of claims 1-12.

16. An electronic device, characterized in that, The electronic device includes: At least one processor; A memory connected to the at least one processor; The memory stores instructions executable by the at least one processor, which implements the image display method according to any one of claims 1 to 12 by executing the instructions stored in the memory.

17. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the image display method according to any one of claims 1 to 12.

Images